Page Comparison

...

pip install --user avocado-framework==57.0
pip install --user avocado_framework_plugin_loader_yaml==57.0
pip install --user avocado_framework_plugin_result_html==57.0
pip install --user avocado_framework_plugin_varianter_yaml_to_mux==57.0
pip install --user clustershell

Avocado Sanity test:

login2(1221)$

...

avocado

...

variants

...

--tree

...

-m

...

daos/src/tests/ftest/pool/attribute.yaml

...

Multiplex

...

tree

...

representation:

...

┗━━

...

run
┣━━ hosts
┣━━ server_config
┗━━ attrtests
┣━━ createmode
┣━━ createset
┣━━ createsize
┣━━ name_handles
┃ ╚══ validlongname
┗━━ value_handles
╚══ validvalue

DAOS patch to run Avocado test on TACC:

Code Block

language	py
title	DAOS_patch
collapse	true

git diff util/server_utils.py
diff --git a/src/tests/ftest/util/server_utils.py b/src/tests/ftest/util/server_utils.py
index 603723e..ddba91a 100644
--- a/src/tests/ftest/util/server_utils.py
+++ b/src/tests/ftest/util/server_utils.py
@@ -80,7 +80,7 @@ class DaosServerConfig(ObjectWithParameters):
             self.targets = BasicParameter(None, 8)
             self.first_core = BasicParameter(None, 0)
             self.nr_xs_helpers = BasicParameter(None, 2)
-            self.fabric_iface = BasicParameter(None, "eth0")
+            self.fabric_iface = BasicParameter(None, "ib0")
             self.fabric_iface_port = BasicParameter(None, 31416)
             self.log_mask = BasicParameter(None, "DEBUG,RPC=ERR,MEM=ERR")
             self.log_file = BasicParameter(None, "/tmp/server.log")
@@ -134,9 +134,9 @@ class DaosServerConfig(ObjectWithParameters):
             #   bdev_list: [/tmp/daos-bdev] - generate nvme.conf as follows:
             #       [AIO]
             #       AIO /tmp/aiofile AIO1 4096
-            self.scm_mount = BasicParameter(None, "/mnt/daos")
+            self.scm_mount = BasicParameter(None, "/dev/shm")
             self.scm_class = BasicParameter(None, "ram")
-            self.scm_size = BasicParameter(None, 6)
+            self.scm_size = BasicParameter(None, 90)
             self.scm_list = BasicParameter(None)
             self.bdev_class = BasicParameter(None)
             self.bdev_list = BasicParameter(None)
@@ -150,8 +150,8 @@ class DaosServerConfig(ObjectWithParameters):
         # Parameters
         self.name = BasicParameter(None, "daos_server")
         self.port = BasicParameter(None, 10001)
-        self.provider = BasicParameter(None, "ofi+sockets")
-        self.socket_dir = BasicParameter(None)          # /tmp/daos_sockets
+        self.provider = BasicParameter(None, "ofi+psm2")
+        self.socket_dir = BasicParameter(None, "/tmp/daos_sockets")          # /tmp/daos_sockets
         self.nr_hugepages = BasicParameter(None, 4096)
         self.control_log_mask = BasicParameter(None, "DEBUG")
         self.control_log_file = BasicParameter(None, "/tmp/daos_control.log")
c455-041[knl](1040)$ git diff util/agent_utils.py
diff --git a/src/tests/ftest/util/agent_utils.py b/src/tests/ftest/util/agent_utils.py
index d8520d0..e5bbe95 100755
--- a/src/tests/ftest/util/agent_utils.py
+++ b/src/tests/ftest/util/agent_utils.py
@@ -70,8 +70,8 @@ def run_agent(basepath, server_list, client_list=None):

     # Verify the domain socket directory is present and owned by this user
     file_checks = (
-        ("Server", server_list, "/var/run/daos_server"),
-        ("Client", client_list, "/var/run/daos_agent"),
+        ("Server", server_list, "/tmp/daos_sockets"),
+        ("Client", client_list, "/tmp/daos_agent"),
     )
     for host_type, host_list, directory in file_checks:
         status, nodeset = check_file_exists(host_list, directory, user)
@@ -88,7 +88,8 @@ def run_agent(basepath, server_list, client_list=None):
     for client in client_list:
         sessions[client] = subprocess.Popen(
             ["ssh", client, "-o ConnectTimeout=10",
-             "{} -i".format(daos_agent_bin)],
+             "{} -i".format(daos_agent_bin),
+             "-s /tmp/daos_agent"],
             stdout=subprocess.PIPE,
             stderr=subprocess.STDOUT
         )
c455-041[knl](1041)$

Avocado test run:

Code Block

language	py
title	Avocado simplecreate functional test
collapse	true

c455-041[knl](1025)$ ./launch.py simplecreate
Arguments: Namespace(archive=False, clean=False, discard=False, include_localhost=False, list=False, rename=False, sparse=False, tags=['simplecreate'], test_clients=None, test_servers=None)
Running avocado list --paginator off --filter-by-tags=simplecreate ./ | sed -ne '/INSTRUMENTED/s/.* \([^:]*\):.*/\1/p' | uniq
Detected tests:
./pool/simple_create_delete_test.py
Running avocado run --ignore-missing-references on --show-job-log --html-job-result on  --filter-by-tags=simplecreate --mux-yaml ./pool/simple_create_delete_test.yaml -- ./pool/simple_create_delete_test.py
found extension EntryPoint.parse('journal = avocado.plugins.journal:JournalResult')
found extension EntryPoint.parse('tap = avocado.plugins.tap:TAPResult')
found extension EntryPoint.parse('human = avocado.plugins.human:Human')
found extension EntryPoint.parse('teststmpdir = avocado.plugins.teststmpdir:TestsTmpDir')
found extension EntryPoint.parse('jobscripts = avocado.plugins.jobscripts:JobScripts')
found extension EntryPoint.parse('human = avocado.plugins.human:HumanJob')
found extension EntryPoint.parse('yaml_to_mux = avocado_varianter_yaml_to_mux:YamlToMux')
File /etc/avocado/sysinfo/commands does not exist.
File /etc/avocado/sysinfo/files does not exist.
File /etc/avocado/sysinfo/profilers does not exist.
Journalctl collection failed: Command 'journalctl --quiet --lines 1 --output json' failed (rc=1)
Command line: /home1/06739/samirrav/.local/bin/avocado run --ignore-missing-references on --show-job-log --html-job-result on --filter-by-tags=simplecreate --mux-yaml ./pool/simple_create_delete_test.yaml -- ./pool/simple_create_delete_test.py

Avocado version: 57.0

Config files read (in order):
/home1/06739/samirrav/.local/lib/python2.7/site-packages/etc/avocado/avocado.conf
/home1/06739/samirrav/.local/lib/python2.7/site-packages/etc/avocado/conf.d/gdb.conf
/home1/06739/samirrav/.config/avocado/avocado.conf

Avocado config:
Section.Key                             Value
datadir.paths.base_dir                  /var/lib/avocado
datadir.paths.test_dir                  /usr/share/avocado/tests
datadir.paths.data_dir                  /var/lib/avocado/data
datadir.paths.logs_dir                  ~/avocado/job-results
sysinfo.collect.enabled                 True
sysinfo.collect.commands_timeout        -1
sysinfo.collect.installed_packages      False
sysinfo.collect.profiler                False
sysinfo.collect.locale                  C
sysinfo.collect.per_test                False
sysinfo.collectibles.commands           /etc/avocado/sysinfo/commands
sysinfo.collectibles.files              /etc/avocado/sysinfo/files
sysinfo.collectibles.profilers          /etc/avocado/sysinfo/profilers
runner.output.colored                   True
runner.output.utf8
remoter.behavior.reject_unknown_hosts   False
remoter.behavior.disable_known_hosts    False
job.output.loglevel                     debug
restclient.connection.hostname          localhost
restclient.connection.port              9405
restclient.connection.username
restclient.connection.password
plugins.disable                         []
plugins.skip_broken_plugin_notification []
plugins.loaders                         ['file', '@DEFAULT']
gdb.paths.gdb                           /usr/bin/gdb
gdb.paths.gdbserver                     /usr/bin/gdbserver

Avocado Data Directories:

base     /home1/06739/samirrav/avocado
tests    /home1/06739/samirrav/.local/lib/python2.7/site-packages/examples/tests
data     /home1/06739/samirrav/avocado/data
logs     /home1/06739/samirrav/avocado/job-results/job-2019-11-27T17.18-e54628a

Multiplex tree representation:
 \-- run
      |-- hosts
      |-- server_config
      \-- tests
           |-- modes
           |    \-- modeall
           |-- uids
           |    #== validuid
           |-- gids
           |    #== validgid
           \-- setnames
                #== validsetname

Multiplex variants (1):
Variant hosts-server_config-validgid-modeall-validsetname-validuid-5e55:    /run/hosts, /run/server_config, /run/tests/modes/modeall, /run/tests/uids/validuid, /run/tests/gids/validgid, /run/tests/setnames/validsetname
Temporary dir: /tmp/avocado_1lRF7N

Job ID: e54628a065c7bfca9df317a02bf054d11c0f0bdb

File /etc/avocado/sysinfo/commands does not exist.
File /etc/avocado/sysinfo/files does not exist.
File /etc/avocado/sysinfo/profilers does not exist.
Journalctl collection failed: Command 'journalctl --quiet --lines 1 --output json' failed (rc=1)
PARAMS (key=timeout, path=*, default=None) => 600
START 1-./pool/simple_create_delete_test.py:SimpleCreateDeleteTest.test_create;hosts-server_config-validgid-modeall-validsetname-validuid-5e55
Test metadata:
  filename: /home1/06739/samirrav/daos/src/tests/ftest/pool/simple_create_delete_test.py
Job-ID: job-2019-11-27T17.18-e54628a
Test PID: 112275
DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
PARAMS (key=fault_list, path=/run/faults/*/, default=None) => None
PARAMS (key=name, path=/server_config/, default=daos_server) => 'daos_server'
PARAMS (key=test_machines, path=/run/hosts/*, default=None) => ['c455-043']
PARAMS (key=test_servers, path=/run/hosts/*, default=None) => None
PARAMS (key=test_clients, path=/run/hosts/*, default=None) => None
PARAMS (key=server_count, path=/run/hosts/*, default=None) => None
PARAMS (key=client_count, path=/run/hosts/*, default=None) => None
PARAMS (key=bdev_class, path=/server_config/server/, default=None) => None
PARAMS (key=server_partition, path=/run/hosts/*, default=None) => None
PARAMS (key=client_partition, path=/run/hosts/*, default=None) => None
hostlist_servers:  ['c455-043']
hostlist_clients:  None
[stdout] <<1>>
[stdout]

[stdout] <AGENT> agent started on node c455-041 in 3.00518798828 seconds
[stdout]

[stdout] Creating the server yaml file
[stdout]

PARAMS (key=control_log_file, path=/run/server_config/*, default=/tmp/daos_control.log) => '/tmp/daos_control.log'
PARAMS (key=control_log_mask, path=/run/server_config/*, default=DEBUG) => 'DEBUG'
PARAMS (key=group_name, path=/run/server_config/*, default=None) => None
PARAMS (key=name, path=/run/server_config, default=daos_server) => 'daos_server'
PARAMS (key=nr_hugepages, path=/run/server_config/*, default=4096) => 4096
PARAMS (key=port, path=/run/server_config/*, default=10001) => 10001
PARAMS (key=provider, path=/run/server_config/*, default=ofi+psm2) => 'ofi+psm2'
PARAMS (key=socket_dir, path=/run/server_config/*, default=/tmp/daos_sockets) => '/tmp/daos_sockets'
PARAMS (key=user_name, path=/run/server_config/*, default=None) => None
PARAMS (key=bdev_class, path=/run/server_config/servers/*, default=None) => None
PARAMS (key=bdev_list, path=/run/server_config/servers/*, default=None) => None
PARAMS (key=bdev_number, path=/run/server_config/servers/*, default=None) => None
PARAMS (key=bdev_size, path=/run/server_config/servers/*, default=None) => None
PARAMS (key=env_vars, path=/run/server_config/servers/*, default=['ABT_ENV_MAX_NUM_XSTREAMS=100', 'ABT_MAX_NUM_XSTREAMS=100', 'DAOS_MD_CAP=1024', 'CRT_CTX_SHARE_ADDR=0', 'CRT_TIMEOUT=30', 'FI_SOCKETS_MAX_CONN_RETRY=1', 'FI_SOCKETS_CONN_TIMEOUT=2000']) => ['ABT_ENV_MAX_NUM_XSTREAMS=100', 'ABT_MAX_NUM_XSTREAMS=100', 'DAOS_MD_CAP=1024', 'CRT_CTX_SHARE_ADDR=0', 'CRT_TIMEOUT=30', 'FI_SOCKETS_MAX_CONN_RETRY=1', 'FI_SOCKETS_CONN_TIMEOUT=2000']
PARAMS (key=fabric_iface, path=/run/server_config/servers/*, default=ib0) => 'ib0'
PARAMS (key=fabric_iface_port, path=/run/server_config/servers/*, default=31416) => 31416
PARAMS (key=first_core, path=/run/server_config/servers/*, default=0) => 0
PARAMS (key=log_file, path=/run/server_config/servers/*, default=/tmp/server.log) => '/tmp/server.log'
PARAMS (key=log_mask, path=/run/server_config/servers/*, default=DEBUG,RPC=ERR,MEM=ERR) => 'DEBUG,RPC=ERR,MEM=ERR'
PARAMS (key=nr_xs_helpers, path=/run/server_config/servers/*, default=2) => 2
PARAMS (key=scm_class, path=/run/server_config/servers/*, default=ram) => 'ram'
PARAMS (key=scm_list, path=/run/server_config/servers/*, default=None) => None
PARAMS (key=scm_mount, path=/run/server_config/servers/*, default=/dev/shm) => '/dev/shm'
PARAMS (key=scm_size, path=/run/server_config/servers/*, default=90) => 90
PARAMS (key=targets, path=/run/server_config/servers/*, default=8) => 8
Updated param log_file => None
[stdout] Removing any existing server processes
[stdout]

[stdout] c455-043: failure running 'pkill '(daos_server|daos_io_server)' --signal INT; sleep 5; pkill '(daos_server|daos_io_server)' --signal KILL': rc=1
[stdout]

[stdout] Cleaning the server tmpfs directories
[stdout]

[stdout] Start CMD>>>>/home1/06739/samirrav/daos/opt/ompi/bin/orterun --np 1 --hostfile /tmp/avocado_1lRF7N/1-._pool_simple_create_delete_test.py_SimpleCreateDeleteTest.test_create_hosts-server_config-validgid-modeall-validsetname-validuid-5e55/hostfile91322 --enable-recovery -x PATH /home1/06739/samirrav/daos/install/bin/daos_server --debug --config /home1/06739/samirrav/daos/src/tests/ftest/data/daos_avocado_test.yaml start -i -a /home1/06739/samirrav/daos/install/tmp
[stdout]

[stdout] <SERVER>: Starting Servers

[stdout] DEBUG 17:18:51.697327 netdetect.go:749: Input provider string: ofi+psm2

[stdout] DEBUG 17:18:52.562352 netdetect.go:731: ValidateProviderConfig (device: ofi+psm2, provider ib0) returned error: Device ib0 does not support provider: ofi+psm2

[stdout] DEBUG 17:18:52.562518 main.go:94: DAOS config loaded from /home1/06739/samirrav/daos/src/tests/ftest/data/daos_avocado_test.yaml

[stdout] DEBUG 17:18:52.562684 netdetect.go:749: Input provider string: ofi+psm2

[stdout] DEBUG 17:18:53.406461 netdetect.go:731: ValidateProviderConfig (device: ofi+psm2, provider ib0) returned error: Device ib0 does not support provider: ofi+psm2

[stdout] /home1/06739/samirrav/daos/install/bin/daos_server logging to file /tmp/daos_control.log

[stdout] DEBUG 17:18:53.406969 start.go:147: Switching control log level to DEBUG

[stdout] DEBUG 17:18:53.407483 server.go:52: cfg: &server.Configuration{ControlPort:10001, TransportConfig:(*security.TransportConfig)(0xc00018ccb0), Servers:[]*ioserver.Config{(*ioserver.Config)(0xc000001b00)}, BdevInclude:[]string(nil), BdevExclude:[]string(nil), NrHugepages:4096, ControlLogMask:3, ControlLogFile:"/tmp/daos_control.log", ControlLogJSON:false, UserName:"", GroupName:"", SystemName:"daos_server", SocketDir:"/tmp/daos_sockets", Fabric:ioserver.FabricConfig{Provider:"ofi+psm2", Interface:"", InterfacePort:0, PinnedNumaNode:(*uint)(nil)}, Modules:"", Attach:"/home1/06739/samirrav/daos/install/tmp", AccessPoints:[]string{"localhost"}, FaultPath:"", FaultCb:"", Hyperthreads:false, Path:"/home1/06739/samirrav/daos/src/tests/ftest/data/daos_avocado_test.yaml", ext:(*server.ext)(0xc000170860), NvmeShmID:1363321369, validateProviderFn:(server.networkProviderValidation)(0x5c05b0), validateNUMAFn:(server.networkNUMAValidation)(0x5c1290)}

[stdout] DEBUG 17:18:53.416692 config.go:403: Active config saved to /home1/06739/samirrav/daos/src/tests/ftest/data/.daos_server.active.yml (read-only)

[stdout] DEBUG 17:18:53.417084 bdev.go:197: spdk : bdev_list empty in config, no nvme.conf generated for server

[stdout] Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...

[stdout] [ DPDK EAL parameters: spdk -c 0x1 --file-prefix=spdk1363321369 --base-virtaddr=0x200000000000 --proc-type=auto ]

[stdout] DEBUG 17:18:53.582905 ctl_storage.go:97: Warning, NVMe Setup: SPDK env init, has setup been run?: spdk_env_opts_init: 1

[stdout] ipmctl lib not present

[stdout] DEBUG 17:18:53.583238 ipmctl.go:102: discovered 0 DCPM modules

[stdout] DAOS control server listening on 0.0.0.0:10001

[stdout] DEBUG 17:18:53.585482 superblock.go:107: /dev/shm: checking superblock

[stdout] DEBUG 17:18:53.585813 instance.go:176: /dev/shm: checking formatting

[stdout] DEBUG 17:18:53.586674 instance.go:193: /dev/shm (ram) needs format: false

[stdout] DEBUG 17:18:53.590255 exec.go:104: daos_io_server:0 config: &ioserver.Config{Rank:(*ioserver.Rank)(nil), Modules:"", TargetCount:8, HelperStreamCount:2, ServiceThreadCore:0, SystemName:"daos_server", SocketDir:"/tmp/daos_sockets", AttachInfoPath:"/home1/06739/samirrav/daos/install/tmp", LogMask:"DEBUG,RPC=ERR,MEM=ERR", LogFile:"", Storage:ioserver.StorageConfig{SCM:storage.ScmConfig{MountPoint:"/dev/shm", Class:"ram", RamdiskSize:90, DeviceList:[]string(nil)}, Bdev:storage.BdevConfig{ConfigPath:"", Class:"", DeviceList:[]string(nil), DeviceCount:0, FileSize:0, ShmID:1363321369, VosEnv:"", Hostname:"c455-043.stampede2.tacc.utexas.edu"}}, Fabric:ioserver.FabricConfig{Provider:"ofi+psm2", Interface:"ib0", InterfacePort:31416, PinnedNumaNode:(*uint)(nil)}, EnvVars:[]string{"ABT_ENV_MAX_NUM_XSTREAMS=100", "ABT_MAX_NUM_XSTREAMS=100", "DAOS_MD_CAP=1024", "CRT_CTX_SHARE_ADDR=0", "CRT_TIMEOUT=30", "FI_SOCKETS_MAX_CONN_RETRY=1", "FI_SOCKETS_CONN_TIMEOUT=2000"}, Index:0x0}

[stdout] DEBUG 17:18:53.590648 exec.go:105: daos_io_server:0 args: [-t 8 -x 2 -g daos_server -d /tmp/daos_sockets -a /home1/06739/samirrav/daos/install/tmp -s /dev/shm -i 1363321369 -I 0]

[stdout] DEBUG 17:18:53.590830 exec.go:106: daos_io_server:0 env: [ABT_MAX_NUM_XSTREAMS=100 FI_SOCKETS_CONN_TIMEOUT=2000 OFI_INTERFACE=ib0 ABT_ENV_MAX_NUM_XSTREAMS=100 DAOS_MD_CAP=1024 CRT_CTX_SHARE_ADDR=0 CRT_TIMEOUT=30 FI_SOCKETS_MAX_CONN_RETRY=1 CRT_PHY_ADDR_STR=ofi+psm2 OFI_PORT=31416]

[stdout] Starting I/O server instance 0: /home1/06739/samirrav/daos/install/bin/daos_io_server

[stdout] daos_io_server:0 Using legacy core allocation algorithm

[stdout] DEBUG 17:18:55.781800 instance.go:219: I/O server instance 0 ready: uri:"ofi+psm2://7990b02:0" nctxs:18 drpcListenerSock:"/tmp/daos_sockets/daos_io_server_101035.sock"

[stdout] DEBUG 17:18:55.782517 instance.go:342: start MS

[stdout] Management Service access point started (bootstrapped)

[stdout] daos_io_server:0 DAOS I/O server (v0.6.0) process 101035 started on rank 0 (out of 1) with 8 target, 2 helper XS per target, firstcore 0, host c455-043.stampede2.tacc.utexas.edu.

[stdout]

[stdout] <SERVER> server started and took 10.8743472099 seconds to start
[stdout]

PARAMS (key=mode, path=/run/tests/modes/*, default=None) => [511, 'PASS']
PARAMS (key=uid, path=/run/tests/uids/*, default=860384) => ['valid', 'PASS']
PARAMS (key=gid, path=/run/tests/gids/*, default=814017) => ['valid', 'PASS']
PARAMS (key=setname, path=/run/tests/setnames/*, default=None) => ['daos_server', 'PASS']
Destroying pools
Stopping agents
Stopping servers
[stdout] <SERVER> server stopped
[stdout]

DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
DATA (filename=stdout.expected) => NOT FOUND (data sources: variant, test, file)
DATA (filename=stderr.expected) => NOT FOUND (data sources: variant, test, file)
Not logging /var/log/messages (lack of permissions)
PASS 1-./pool/simple_create_delete_test.py:SimpleCreateDeleteTest.test_create;hosts-server_config-validgid-modeall-validsetname-validuid-5e55

Test results available in /home1/06739/samirrav/avocado/job-results/job-2019-11-27T17.18-e54628a
Total test time: 67s
All avocado tests passed!
c455-041[knl](1026)$

Versions Compared

Old Version 10

New Version Current

Key

Avocado Sanity test:

DAOS patch to run Avocado test on TACC:

Avocado test run: