DAOS-6820 -1.2 Verbs Automated testing status

Single pool works fine but multiple pools fail with NO SPACE issue. looks like test issue 

to container 42B8B01A-DBCF-4328-96B1-4F0275885CE3: Object update returned non-zero. RC: -1007

/scratch/samirrav/verbs_Logs/server/job-2021-03-11T19.11-718b938-server-cpu_usage

/scratch/samirrav/verbs_Logs/server/job-2021-03-04T21.07-c8285fd-server-cpu_usage/ 

Looks like some test issue where IOR failed to write 

[stderr] [1,0]<stderr>:Failed to lookup parent dir
[stderr] [1,0]<stderr>:application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0
[stderr] [1,0]<stderr>:[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=-1
[stderr] [1,0]<stderr>::
[stderr] [1,0]<stderr>:system msg for write_line failure : Bad file descriptor

Looks like test issue 

Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -j system query --verbose' finished with 0 after 0.219604969025s
system_query data: {}
<SERVER> Verifying server states: group=daos_server, hosts=wolf-[118-119]
  Unable to obtain current server state.  Undefined expected server states due to a failure starting the servers.
Writing yaml configuration file /var/tmp/daos_testing/test_daos_server_dmg.yaml
Copying /var/tmp/daos_testing/test_daos_server_dmg.yaml yaml configuration file to /etc/daos/daos_control.yml on ['wolf-79']
Command environment vars:
  None
Running '/usr/bin/dmg -o /etc/daos/daos_control.yml -j system query --verbose'
[stdout] {
[stderr] ERROR: dmg: unable to contact the DAOS Management Service
[stdout]   "response": {
[stdout]     "members": null
[stdout]   },
[stdout]   "error": "unable to contact the DAOS Management Service",
[stdout]   "status": -1025
[stdout] }
Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -j system query --verbose' finished with 1 after 0.1741938591s
Error occurred running '/usr/bin/dmg -o /etc/daos/daos_control.yml -j system query --verbose': Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -j system query --verbose' failed (rc=1)

DAOS-6893

Test issue

Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/avocado/core/test.py", line 667, in _run_avocado
    raise test_exception

TypeError: create_cont() takes exactly 2 arguments (1 given)

Test Issue 

TypeError: create_cont() takes exactly 2 arguments (1 given)

Test issue Server is trying to start on client

"Failed to start servers before format: Error occurred running 'sudo -n systemctl enable daos_server.service' on wolf-79",

Looks like a test issue,

2021-02-27 00:33:16,800 stacktrace       L0039 ERROR|
2021-02-27 00:33:16,800 stacktrace       L0042 ERROR| Reproduced traceback from: /usr/lib/python2.7/site-packages/avocado/core/test.py:596
2021-02-27 00:33:16,800 stacktrace       L0045 ERROR| Traceback (most recent call last):
2021-02-27 00:33:16,800 stacktrace       L0045 ERROR|   File "/usr/lib/daos/TESTING/ftest/control/daos_admin_privileged.py", line 38, in test_daos_admin_format
2021-02-27 00:33:16,801 stacktrace       L0045 ERROR|     file_stats = os.stat("/usr/bin/daos_admin")
2021-02-27 00:33:16,801 stacktrace       L0045 ERROR| OSError: [Errno 2] No such file or directory: '/usr/bin/daos_admin'
2021-02-27 00:33:16,801 stacktrace       L0046 ERROR|

Looks like some test reporting issue

ERROR Test reported status but did not finish -> TestAbortedError: 1-./pool/connect_test.py:ConnectTest.test_connect;hosts-pool-server_config-validsetname-cc29

Test timeout issue so tried increasing the timeout

Test got a pass but saw this error so looks like the test need to be fixed

Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -j pool create --group=samirrav --nsvc=0 --scm-size=134217728 --sys=daos_server --user=samirrav' finished with 0 after 9.0510468483s
## TestFail exception is caught at pool create!
Test was expected to fail, but it passed.

Traceback (most recent call last):
  File "/usr/lib/daos/TESTING/ftest/pool/pool_svc.py", line 43, in test_poolsvc
    self.fail("Test was expected to fail, but it passed.\n")
  File "/usr/lib/python2.7/site-packages/avocado/core/test.py", line 762, in fail
    raise exceptions.TestFail(message)
TestFail: Test was expected to fail, but it passed.

Test issue

Running '/usr/bin/dmg -o /etc/daos/daos_control.yml -j pool create --group=samirrav --nsvc=1 --nvme-size=4.0GiB --scm-size=0.0B --sys=daos_server --user=samirrav'
[stdout] {
[stderr] ERROR: dmg: failed to generate PoolCreate request: can't create pool with 0 SCM
[stdout]   "response": null,
[stdout]   "error": "failed to generate PoolCreate request: can't create pool with 0 SCM",
[stdout]   "status": -1025
[stdout] }

Test issue

Running '/usr/bin/dmg -o /etc/daos/daos_control.yml -j pool create --group=samirrav --nsvc=1 --nvme-size=8.0GiB --ranks=0 --scm-size=0.0B --sys=daos_server --user=samirrav'
[stdout] {
[stderr] ERROR: dmg: failed to generate PoolCreate request: can't create pool with 0 SCM
[stdout]   "response": null,
[stdout]   "error": "failed to generate PoolCreate request: can't create pool with 0 SCM",
[stdout]   "status": -1025
[stdout] }

Testtimeout issue

2021-03-05 22:18:15,813 stacktrace       L0045 ERROR|   File "/usr/lib/python2.7/site-packages/avocado/core/runner.py", line 312, in sigterm_handler
2021-03-05 22:18:15,813 stacktrace       L0045 ERROR|     raise RuntimeError("Test interrupted by SIGTERM")
2021-03-05 22:18:15,813 stacktrace       L0045 ERROR| RuntimeError: Test interrupted by SIGTERM

Looks like test issue

2021-03-05 22:41:57,606 process          L0389 INFO | Running '/usr/bin/dmg -o /etc/daos/daos_control.yml -j pool create --group=samirrav --ranks=2 --scm-size=1G --user=samirrav'
2021-03-05 22:41:57,832 process          L0479 DEBUG| [stdout] {
2021-03-05 22:41:57,832 process          L0479 DEBUG| [stderr] ERROR: dmg: pool create failed: server: code = 644 description = "pool request contains invalid ranks: 2"
2021-03-05 22:41:57,833 process          L0479 DEBUG| [stdout]   "response": null,
2021-03-05 22:41:57,833 process          L0479 DEBUG| [stdout]   "error": "pool create failed: server: code = 644 description = \"pool request contains invalid ranks: 2\"",
2021-03-05 22:41:57,833 process          L0479 DEBUG| [stdout]   "status": -1025
2021-03-05 22:41:57,833 process          L0479 DEBUG| [stderr] ERROR: dmg: server: code = 644 resolution = "retry the request with a valid set of ranks"
2021-03-05 22:41:57,834 process          L0479 DEBUG| [stdout] }
2021-03-05 22:41:57,836 process          L0499 INFO | Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -j pool create --group=samirrav --ranks=2 --scm-size=1G --user=samirrav' finished with 1 after 0.226467132568s
2021-03-05 22:41:57,837 output           L0655 DEBUG| Error occurred running '/usr/bin/dmg -o /etc/daos/daos_control.yml -j pool create --group=samirrav --ranks=2 --scm-size=1G --user=samirrav': Command '/usr/bin/dmg -o /etc/daos/daos_control.yml -j pool create --group=samirrav --ranks=2 --scm-size=1G --user=samirrav' failed (rc=1)

Test issue

Reproduced traceback from: /usr/lib/python2.7/site-packages/avocado/core/test.py:596
Traceback (most recent call last):
  File "/usr/lib/daos/TESTING/ftest/pool/bad_evict.py", line 74, in test_evict
    pool.pool.evict()
AttributeError: 'DaosPool' object has no attribute 'evict'

Looks like change in behavior so test code needs to be updated

2021-03-05 23:31:02,976 stacktrace       L0039 ERROR|
2021-03-05 23:31:02,977 stacktrace       L0042 ERROR| Reproduced traceback from: /usr/lib/python2.7/site-packages/avocado/core/test.py:596
2021-03-05 23:31:02,977 stacktrace       L0045 ERROR| Traceback (most recent call last):
2021-03-05 23:31:02,977 stacktrace       L0045 ERROR|   File "/usr/lib/daos/TESTING/ftest/pool/storage_ratio.py", line 68, in test_storage_ratio
2021-03-05 23:31:02,977 stacktrace       L0045 ERROR|     .format(tests[key], tests[key][2]))
2021-03-05 23:31:02,977 stacktrace       L0045 ERROR|   File "/usr/lib/python2.7/site-packages/avocado/core/test.py", line 762, in fail
2021-03-05 23:31:02,977 stacktrace       L0045 ERROR|     raise exceptions.TestFail(message)
2021-03-05 23:31:02,977 stacktrace       L0045 ERROR| TestFail: Pool Creation ['1G', '200G', 'WARNING'] Suppose to WARNING

There looks like multiple test issues:

1.    servers: !mux is not going to work now
       1_server:
         test_servers:
           - server-A
       4_servers:
         test_servers:
           - server-A
           - server-B
           - server-C
           - server-D
2. "RP_2GX" will not be able to run with a single server

 Looks like the Test issue

-> mean_mib <type 'int'>: 3
 -> write_x <type 'int'>: 3
*** TEARDOWN called after test completion: elapsed time: 340.311693907 seconds ***

Added timeout between pool query. For logs, check DAOS-6580.

try:

if display_space:
time.sleep(5)
pool.display_pool_daos_space()

Local env issue

[stderr] [wolf-81.wolf.hpdd.intel.com:38374] PMIX ERROR: BAD-PARAM in file event/pmix_event_notification.c at line 848

Test Issue

Reproduced traceback from: /usr/lib/python2.7/site-packages/avocado/core/test.py:596
Traceback (most recent call last):
  File "/usr/lib/daos/TESTING/ftest/io/io_aggregation.py", line 82, in test_ioaggregation
    self.run_ior_with_pool()
  File "/usr/lib/daos/TESTING/ftest/util/ior_test_base.py", line 139, in run_ior_with_pool
    self.update_ior_cmd_with_pool(create_cont)
  File "/usr/lib/daos/TESTING/ftest/util/ior_test_base.py", line 176, in update_ior_cmd_with_pool
    self.create_pool()
  File "/usr/lib/daos/TESTING/ftest/util/ior_test_base.py", line 66, in create_pool
    self.pool.create()
  File "/usr/lib/python2.7/site-packages/avocado/core/decorators.py", line 51, in wrap
    raise core_exceptions.TestFail(str(details))
TestFail: Error: control method API not supported for create()

Test issue

2021-03-08 19:58:14,242 stacktrace       L0039 ERROR|
2021-03-08 19:58:14,242 stacktrace       L0042 ERROR| Reproduced traceback from: /usr/lib/python2.7/site-packages/avocado/core/test.py:596
2021-03-08 19:58:14,243 stacktrace       L0045 ERROR| Traceback (most recent call last):
2021-03-08 19:58:14,243 stacktrace       L0045 ERROR|   File "/usr/lib/daos/TESTING/ftest/io/dfuse_sparse_file.py", line 49, in test_dfusesparsefile
2021-03-08 19:58:14,243 stacktrace       L0045 ERROR|     self.create_cont()
2021-03-08 19:58:14,243 stacktrace       L0045 ERROR| TypeError: create_cont() takes exactly 2 arguments (1 given)
2021-03-08 19:58:14,243 stacktrace       L0046 ERROR|
2021-03-08 19:58:14,243 test             L0601 DEBUG| Local variables:

Test issue 

[stdout] {
[stderr] ERROR: dmg: pool create failed: DAOS error (-1007): DER_NOSPACE
[stdout]   "response": null,
[stdout]   "error": "pool create failed: DAOS error (-1007): DER_NOSPACE",
[stdout]   "status": -1007
[stdout] }

Test issue

Reproduced traceback from: /usr/lib/python2.7/site-packages/avocado/core/test.py:596
Traceback (most recent call last):
  File "/usr/lib/daos/TESTING/ftest/io/large_file_count.py", line 64, in test_largefilecount
    self.container.destroy()
AttributeError: 'NoneType' object has no attribute 'destroy'

Test time issue

Command '/usr/bin/dmg -o /etc/daos/daos_control.yml pool query --pool=B33BF608-6A5B-4283-A9D9-E4B3898107C9' finished with 0 after 32.22884202s
Pool query is responsive

Reproduced traceback from: /usr/lib/python2.7/site-packages/avocado/core/test.py:596
Traceback (most recent call last):
  File "/usr/lib/daos/TESTING/ftest/rebuild/rebuild_widely_striped.py", line 82, in test_rebuild_widely_striped
    self.pool.wait_for_rebuild(False, interval=1)
  File "/usr/lib/daos/TESTING/ftest/util/test_utils_pool.py", line 504, in wait_for_rebuild
    format(self.pool_query_timeout.value))
DaosTestError: TIMEOUT detected after 30 seconds of pool query. This timeout can be adjusted via the 'pool/pool_query_timeout' test yaml parameter.

Test time issue tried after increasing got timeout for pool query

Command '/usr/bin/dmg -o /etc/daos/daos_control.yml pool query --pool=75AC2E47-139C-437B-B600-FCF1471122BC' finished with 0 after 32.2416799068s
Pool query is responsive

Reproduced traceback from: /usr/lib/python2.7/site-packages/avocado/core/test.py:596
Traceback (most recent call last):
  File "/usr/lib/daos/TESTING/ftest/rebuild/pool_destroy_rebuild.py", line 76, in test_pool_destroy_with_io
    self.pool.wait_for_rebuild(True, interval=1)
  File "/usr/lib/daos/TESTING/ftest/util/test_utils_pool.py", line 504, in wait_for_rebuild
    format(self.pool_query_timeout.value))
DaosTestError: TIMEOUT detected after 15 seconds of pool query. This timeout can be adjusted via the 'pool/pool_query_timeout' test yaml parameter.

Test Failure looks like the existing test defect 

Observed the same failure as Ravi mention and looks like most defects are open and issue caused because of DAOS-5945

Ravi's input: When I ran all the tests, it was taking long time with various tests failing with timeout. So ran a single pool test and created DAOS-6549. DAOS-6549 is resolved stating DAOS-5945 is the root-cause for the problem.

/scratch/samirrav/verbs_Logs/daos_test/job-2021-03-11T20.30-cd51ab4-daos_test-daos_core_test-rebuild/

On master -/scratch/samirrav/verbs_Logs/daos_test/job-2021-03-17T23.41-3423e9c-daos_test-daos_core_test-rebuild

Only 23-24-25 - /scratch/samirrav/verbs_Logs/daos_test/job-2021-03-18T00.27-5f8e7df-daos_test-daos_core_test-rebuild

Tried running it again on master and now rebuild test 23/24/25 failed. Looks like test 23 failed but 24/25 did not run because of the previous fail so seems test issue? Tried running it 23,24,25 again and all Three got passed.

Observed timeout for test_rebuild_27 to be debug

Ravi's input: When I ran all the tests, it was taking a long time with various tests failing with timeout. So ran rebuild tests (12-15) test and created DAOS-6610. DAOS-6610 is resolved stating DAOS-5945 is the root-cause for the problem.