Test Configuration

Tip of master, commit 7cffa6f494db6d95df1f08033ccd5cbfe36f93eb

All tests run with ofi+psm2, ib0.

daos_test: Run with 8 server (boro-[3-9],boro-58), 2 client (boro-[59-60]). Killed servers, cleaned /mnt/daos in between runs listed below.

Tests requiring pool to be created via dmg used 4GB pool. These used boro-59 as client.

mpich tests used boro-3 as server, boro-59 as client, with a 1GB pool.

Test Results

daos_test

Separate runs with cleanup in between:

-mpcCiAeoRd - PASS
-r - PASS

daosperf

1K Records

CREDITS=1

[sdwillso@boro-59 ~]$ CREDITS=1 ./daos_m/src/tests/daos_perf.sh daos 200 1000 1K
+ /home/sdwillso/daos_m/opt/ompi/bin/orterun -quiet --hostfile /home/sdwillso/scripts/host.cli.1 --ompi-server file:/home/sdwillso/scripts/uri.txt -x DD_SUBSYS= -x DD_MASK= -x D_LOG_FILE=/tmp/daos_perf.log /home/sdwillso/daos_m/install/bin/daos_perf -T daos -P 2G -d 1 -a 200 -r 1000 -s 1K -C 1 -t -z
Test :
	DAOS (full stack)
Parameters :
	pool size     : 2048 MB
	credits       : 1 (sync I/O for -ve)
	obj_per_cont  : 1 x 8 (procs)
	dkey_per_obj  : 1
	akey_per_dkey : 200
	recx_per_akey : 1000
	value type    : single
	value size    : 1024
	zero copy     : yes
	overwrite     : yes
	VOS file      : <NULL>
Started...
update successfully completed:
	duration : 11.453220  sec
	bandwith : 136.425    MB/sec
	rate     : 139698.71  IO/sec
	latency  : 7.158      us (nonsense if credits > 1)
Duration across processes:
MAX duration : 11.449782  sec
MIN duration : 6.149746   sec
Average duration : 9.161313   sec

CREDITS=8

[sdwillso@boro-59 ~]$ CREDITS=8 ./daos_m/src/tests/daos_perf.sh daos 200 1000 1K
+ /home/sdwillso/daos_m/opt/ompi/bin/orterun -quiet --hostfile /home/sdwillso/scripts/host.cli.1 --ompi-server file:/home/sdwillso/scripts/uri.txt -x DD_SUBSYS= -x DD_MASK= -x D_LOG_FILE=/tmp/daos_perf.log /home/sdwillso/daos_m/install/bin/daos_perf -T daos -P 2G -d 1 -a 200 -r 1000 -s 1K -C 8 -t -z
Test :
	DAOS (full stack)
Parameters :
	pool size     : 2048 MB
	credits       : 8 (sync I/O for -ve)
	obj_per_cont  : 1 x 8 (procs)
	dkey_per_obj  : 1
	akey_per_dkey : 200
	recx_per_akey : 1000
	value type    : single
	value size    : 1024
	zero copy     : yes
	overwrite     : yes
	VOS file      : <NULL>
Started...
update successfully completed:
	duration : 9.665754   sec
	bandwith : 161.653    MB/sec
	rate     : 165532.87  IO/sec
	latency  : 6.041      us (nonsense if credits > 1)
Duration across processes:
MAX duration : 9.664282   sec
MIN duration : 3.791001   sec
Average duration : 7.475405   sec

4K Records

CREDITS=1

[sdwillso@boro-59 ~]$ CREDITS=1 ./daos_m/src/tests/daos_perf.sh daos 200 1000 4K
+ /home/sdwillso/daos_m/opt/ompi/bin/orterun -quiet --hostfile /home/sdwillso/scripts/host.cli.1 --ompi-server file:/home/sdwillso/scripts/uri.txt -x DD_SUBSYS= -x DD_MASK= -x D_LOG_FILE=/tmp/daos_perf.log /home/sdwillso/daos_m/install/bin/daos_perf -T daos -P 2G -d 1 -a 200 -r 1000 -s 4K -C 1 -t -z
Test :
	DAOS (full stack)
Parameters :
	pool size     : 2048 MB
	credits       : 1 (sync I/O for -ve)
	obj_per_cont  : 1 x 8 (procs)
	dkey_per_obj  : 1
	akey_per_dkey : 200
	recx_per_akey : 1000
	value type    : single
	value size    : 4096
	zero copy     : yes
	overwrite     : yes
	VOS file      : <NULL>
Started...
update successfully completed:
	duration : 14.699587  sec
	bandwith : 425.182    MB/sec
	rate     : 108846.60  IO/sec
	latency  : 9.187      us (nonsense if credits > 1)
Duration across processes:
MAX duration : 14.699079  sec
MIN duration : 6.184355   sec
Average duration : 10.489390  sec

IOR, single client 4GB pool

[sdwillso@boro-59 ~]$ orterun -np 1 --mca mtl ^psm2,ofi  --ompi-server file:~/scripts/uri.txt ior -v  -i 5 -a DAOS -w -o `uuidgen` -b 10g -t 1m -O daospool=57de6d7e-313d-467c-9cf9-817c7261a3f1,daosrecordsize=1m,daosstripesize=1m,daosstripecount=1024,daosaios=16,daosobjectclass=LARGE,daosPoolSvc=1,daosepoch=1
IOR-3.0.1: MPI Coordinated Test of Parallel I/O

Began: Tue May 15 18:23:26 2018
Command line used: ior -v -i 5 -a DAOS -w -o 4ae6dedf-f455-4100-9c49-6c36f18d205a -b 10g -t 1m -O daospool=57de6d7e-313d-467c-9cf9-817c7261a3f1,daosrecordsize=1m,daosstripesize=1m,daosstripecount=1024,daosaios=16,daosobjectclass=LARGE,daosPoolSvc=1,daosepoch=1
Machine: Linux boro-59.boro.hpdd.intel.com
Start time skew across all tasks: 0.00 sec

Test 0 started: Tue May 15 18:23:26 2018
Path: /home/sdwillso
FS: 3.8 TiB   Used FS: 9.0%   Inodes: 250.0 Mi   Used Inodes: 1.8%
Participating tasks: 1
[0] WARNING: USING daosStripeMax CAUSES READS TO RETURN INVALID DATA
Summary:
	api                = DAOS
	test filename      = 4ae6dedf-f455-4100-9c49-6c36f18d205a
	access             = single-shared-file, independent
	pattern            = segmented (1 segment)
	ordering in a file = sequential offsets
	ordering inter file= no tasks offsets
	clients            = 1 (1 per node)
	repetitions        = 5
	xfersize           = 1 MiB
	blocksize          = 10 GiB
	aggregate filesize = 10 GiB

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Tue May 15 18:23:26 2018
write     6122       10485760   1024.00    0.001180   1.67       0.003044   1.67       0   
remove    -          -          -          -          -          -          0.008123   0   
Commencing write performance test: Tue May 15 18:23:28 2018
write     6047       10485760   1024.00    0.000790   1.69       0.003101   1.69       1   
remove    -          -          -          -          -          -          0.007797   1   
Commencing write performance test: Tue May 15 18:23:30 2018
write     5787       10485760   1024.00    0.000708   1.77       0.003190   1.77       2   
remove    -          -          -          -          -          -          0.007731   2   
Commencing write performance test: Tue May 15 18:23:32 2018
write     6078       10485760   1024.00    0.000711   1.68       0.002450   1.68       3   
remove    -          -          -          -          -          -          0.007542   3   
Commencing write performance test: Tue May 15 18:23:33 2018
write     6120       10485760   1024.00    0.000705   1.67       0.002252   1.67       4   
remove    -          -          -          -          -          -          0.007559   4   

Max Write: 6122.37 MiB/sec (6419.77 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write        6122.37    5786.92    6030.81     125.16    1.69870 0 1 1 5 0 0 1 0 0 1 10737418240 1048576 10737418240 DAOS 0

IOR, 2 client 10GB pool, data verification enabled

[sdwillso@boro-59 ~]$ orterun -np 1 --hostfile ~/hostlists/daos_client_hostlist --mca mtl ^psm2,ofi  --ompi-server file:~/scripts/uri.txt ior -W -v  -i 5 -a DAOS -w -o `uuidgen` -b 10g -t 1m -O daospool=2b8ce7c0-5bba-4fc0-909b-6d968b252b93,daosrecordsize=1m,daosstripesize=1m,daosstripecount=1024,daosaios=16,daosobjectclass=LARGE,daosPoolSvc=1,daosepoch=1
IOR-3.0.1: MPI Coordinated Test of Parallel I/O

Began: Tue May 15 22:16:04 2018
Command line used: ior -W -v -i 5 -a DAOS -w -o f6b0d019-98f2-4c49-bd25-25a0d66c6c2f -b 10g -t 1m -O daospool=2b8ce7c0-5bba-4fc0-909b-6d968b252b93,daosrecordsize=1m,daosstripesize=1m,daosstripecount=1024,daosaios=16,daosobjectclass=LARGE,daosPoolSvc=1,daosepoch=1
Machine: Linux boro-59.boro.hpdd.intel.com
Start time skew across all tasks: 0.00 sec

Test 0 started: Tue May 15 22:16:04 2018
Path: /home/sdwillso
FS: 3.8 TiB   Used FS: 9.0%   Inodes: 250.0 Mi   Used Inodes: 1.8%
Participating tasks: 1
[0] WARNING: USING daosStripeMax CAUSES READS TO RETURN INVALID DATA
Summary:
	api                = DAOS
	test filename      = f6b0d019-98f2-4c49-bd25-25a0d66c6c2f
	access             = single-shared-file, independent
	pattern            = segmented (1 segment)
	ordering in a file = sequential offsets
	ordering inter file= no tasks offsets
	clients            = 1 (1 per node)
	repetitions        = 5
	xfersize           = 1 MiB
	blocksize          = 10 GiB
	aggregate filesize = 10 GiB

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Tue May 15 22:16:04 2018
write     3724       10485760   1024.00    0.001198   2.75       0.002832   2.75       0   
Verifying contents of the file(s) just written.
Tue May 15 22:16:07 2018

remove    -          -          -          -          -          -          0.007806   0   
Commencing write performance test: Tue May 15 22:16:15 2018
write     3769       10485760   1024.00    0.000693   2.71       0.003378   2.72       1   
Verifying contents of the file(s) just written.
Tue May 15 22:16:17 2018

remove    -          -          -          -          -          -          0.007733   1   
Commencing write performance test: Tue May 15 22:16:25 2018
write     3749       10485760   1024.00    0.000684   2.73       0.002396   2.73       2   
Verifying contents of the file(s) just written.
Tue May 15 22:16:28 2018

remove    -          -          -          -          -          -          0.007591   2   
Commencing write performance test: Tue May 15 22:16:36 2018
write     3774       10485760   1024.00    0.000679   2.71       0.002356   2.71       3   
Verifying contents of the file(s) just written.
Tue May 15 22:16:39 2018

remove    -          -          -          -          -          -          0.007547   3   
Commencing write performance test: Tue May 15 22:16:47 2018
write     3784       10485760   1024.00    0.000760   2.70       0.002375   2.71       4   
Verifying contents of the file(s) just written.
Tue May 15 22:16:49 2018

remove    -          -          -          -          -          -          0.007593   4   

Max Write: 3784.03 MiB/sec (3967.84 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write        3784.03    3723.54    3760.10      21.55    2.72342 0 1 1 5 0 0 1 0 0 1 10737418240 1048576 10737418240 DAOS 0

daos_bench

kv-idx-update

Time: 66.753337 seconds (14980.524402 ops per second)

[sdwillso@boro-59 ~]$ orterun -np 1 --mca mtl ^psm2,ofi  --ompi-server file:~/scripts/uri.txt daosbench --test=kv-idx-update --testid=1 --svc=1 --dpool=357aa259-1ab1-4fe3-9125-71bab9ed9139 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Tue May 15 18:27:17 2018
=================================
===============================
Test Setup
---------------
Test: kv-idx-update
DAOS pool :357aa259-1ab1-4fe3-9125-71bab9ed9139
DAOS container :7e5c5fb5-b197-4752-8e56-a45ffabc9005
Value buffer size: 64
Number of processes: 1
Number of indexes/process: 1000000
Number of asynchronous I/O: 32
===============================
kv-idx-update
Time: 66.753337 seconds (14980.524402 ops per second)

kv-dkey-update

Time: 0.011910 seconds (8396.287980 ops per second)

[sdwillso@boro-59 ~]$ orterun -np 1 --mca mtl ^psm2,ofi  --ompi-server file:~/scripts/uri.txt daosbench --test=kv-dkey-update --testid=1 --svc=1 --dpool=4b5b1db0-245e-4037-aa61-47cb5006cace --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Tue May 15 18:36:16 2018
=================================
===============================
Test Setup
---------------
Test: kv-dkey-update
DAOS pool :4b5b1db0-245e-4037-aa61-47cb5006cace
DAOS container :10bb507e-cd78-4d38-9865-961c10480a37
Value buffer size: 64
Number of processes: 1
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-dkey-update
Time: 0.011910 seconds (8396.287980 ops per second)

kv-akey-update

Time: 0.010629 seconds (9407.848418 ops per second)

[sdwillso@boro-59 ~]$ orterun -np 1 --mca mtl ^psm2,ofi  --ompi-server file:~/scripts/uri.txt daosbench --test=kv-akey-update --testid=1 --svc=1 --dpool=c67bcdbd-36ec-4e00-857f-204df99f0646 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Tue May 15 18:39:29 2018
=================================
===============================
Test Setup
---------------
Test: kv-akey-update
DAOS pool :c67bcdbd-36ec-4e00-857f-204df99f0646
DAOS container :288e9ad9-42e6-4016-9284-93d2a011f1f3
Value buffer size: 64
Number of processes: 1
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-akey-update
Time: 0.010629 seconds (9407.848418 ops per second)

kv-dkey-fetch

Time: 0.006670 seconds (14993.068912 ops per second)

[sdwillso@boro-59 ~]$ orterun -np 1 --mca mtl ^psm2,ofi  --ompi-server file:~/scripts/uri.txt daosbench --test=kv-dkey-fetch --testid=1 --svc=1 --dpool=f7849c43-4508-4fc3-a866-de3a998cd3a7 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Tue May 15 18:41:31 2018
=================================
===============================
Test Setup
---------------
Test: kv-dkey-fetch
DAOS pool :f7849c43-4508-4fc3-a866-de3a998cd3a7
DAOS container :318250b7-e15b-442c-b746-29e7b2cc7229
Value buffer size: 64
Number of processes: 1
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-dkey-fetch
Time: 0.006670 seconds (14993.068912 ops per second)

kv-akey-fetch

Time: 0.006212 seconds (16098.672065 ops per second)

[sdwillso@boro-59 ~]$ orterun -np 1 --mca mtl ^psm2,ofi  --ompi-server file:~/scripts/uri.txt daosbench --test=kv-akey-fetch --testid=1 --svc=1 --dpool=5999fb3b-2f4d-4c3f-b7d5-79a9be8d13a4 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Tue May 15 18:43:19 2018
=================================
===============================
Test Setup
---------------
Test: kv-akey-fetch
DAOS pool :5999fb3b-2f4d-4c3f-b7d5-79a9be8d13a4
DAOS container :b3291b9d-a3bc-4d3e-b250-e21e3d8597fa
Value buffer size: 64
Number of processes: 1
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-akey-fetch
Time: 0.006212 seconds (16098.672065 ops per second)

mpich tests

Results:

No Errors on all tests for ~50% of attempts, segfault in daos_server in 50% of attempts. No ticket has been filed yet, though Mohamad has begun debug. For convenience, the text of his email exchange to the daos-devel-request mailing list is pasted here:

Stephen ran into a server segfault when testing mpich. I verified and replicated the segfault, and it seems it has to do with ofi/PSM2 (trace below).
I didn’t see that segfault before, and it doesn’t occur all the time. I just ran the mpi-io tests and replicated once on the first run an another time on the second run, so it’s easy to replicate although not always.
 
Here are my env variables:
export CRT_PHY_ADDR_STR="ofi+psm2"
export OFI_INTERFACE=ib0
export OFI_PORT=26850
export FI_PSM2_NAME_SERVER=1
export PSM2_MULTI_EP=1
export FI_SOCKETS_MAX_CONN_RETRY=1
export CRT_CTX_SHARE_ADDR=1
export CRT_CTX_NUM=8
export D_LOG_MASK=CRIT
export DD_SUBSYS=0
 
On the client I have the same as above but I add this (CRT_CTX_NUM is overwritten):
export DAOS_SINGLETON_CLI=1
export CRT_ATTACH_INFO_PATH=~
export FI_PSM2_DISCONNECT=1
export CRT_CTX_NUM=2
 
From the trace it seems we run into this on a pool connect?
Again I never run into this before, so seems like a regression at the server side.
 
Does this look familiar?
 
Segfault trace:
#0  0x00007ffff3f0cd0f in psmx2_write_generic ()
   from /scratch/mschaara/DEPS/ofi/lib/libfabric.so.1
#1  0x00007ffff3f0e458 in psmx2_writev ()
   from /scratch/mschaara/DEPS/ofi/lib/libfabric.so.1
#2  0x00007ffff536f35e in fi_writev (context=0x7fffcd23a928, key=2, 
    addr=<optimized out>, dest_addr=3940649673949191, count=1, 
    desc=0x7fffd81e0538, iov=0x7fffd81e0540, ep=0x7fde60)
---Type <return> to continue, or q <return> to quit---
    at /scratch/mschaara/DEPS/ofi/include/rdma/fi_rma.h:130
#3  na_ofi_put (na_class=0x6d88d0, context=0x7fffcc02d510, 
    callback=<optimized out>, arg=<optimized out>, 
    local_mem_handle=<optimized out>, local_offset=<optimized out>, 
    remote_mem_handle=0x7fffcd23e900, remote_offset=0, length=64, 
    remote_addr=0x7fffcd23d8a0, remote_id=0 '\000', op_id=0x7fffcd23c9f0)
    at /home/mschaara/source/deps_daos/daos_m/_build.external/mercury/src/na/na_ofi.c:3642
#4  0x00007ffff558ddd1 in hg_bulk_transfer_pieces (
    na_bulk_op=na_bulk_op@entry=0x7ffff558dbf0 <hg_bulk_na_put>, 
    origin_addr=origin_addr@entry=0x7fffcd23d8a0, origin_id=0 '\000', 
    hg_bulk_origin=hg_bulk_origin@entry=0x7fffcd239840, 
    origin_segment_start_index=origin_segment_start_index@entry=0, 
    origin_segment_start_offset=origin_segment_start_offset@entry=0, 
    hg_bulk_local=hg_bulk_local@entry=0x7fffcd23a7b0, 
    local_segment_start_index=local_segment_start_index@entry=0, 
    local_segment_start_offset=local_segment_start_offset@entry=0, 
    size=size@entry=64, scatter_gather=scatter_gather@entry=0 '\000', 
    hg_bulk_op_id=hg_bulk_op_id@entry=0x7fffcd23e760, 
    na_op_count=na_op_count@entry=0x0, use_sm=0 '\000')
    at /home/mschaara/source/deps_daos/daos_m/_build.external/mercury/src/mercury_bulk.c:784
#5  0x00007ffff558fb98 in hg_bulk_transfer (op_id=0x7fffd81e0838, 
---Type <return> to continue, or q <return> to quit---
    op_id@entry=0x40, size=64, size@entry=0, local_offset=<optimized out>, 
    hg_bulk_local=0x7fffcd23a7b0, hg_bulk_local@entry=0x0, 
    origin_offset=<optimized out>, hg_bulk_origin=0x7fffcd239840, 
    origin_id=<optimized out>, origin_addr=<optimized out>, 
    op=<optimized out>, arg=0x7fffcd23eaf0, 
    callback=0x7ffff7754080 <crt_hg_bulk_transfer_cb>, 
    context=<optimized out>)
    at /home/mschaara/source/deps_daos/daos_m/_build.external/mercury/src/mercury_bulk.c:955
#6  HG_Bulk_transfer_id (context=<optimized out>, 
    callback=callback@entry=0x7ffff7754080 <crt_hg_bulk_transfer_cb>, 
    arg=arg@entry=0x7fffcd23eaf0, op=<optimized out>, 
    origin_addr=<optimized out>, origin_id=origin_id@entry=0 '\000', 
    origin_handle=0x7fffcd239840, origin_offset=origin_offset@entry=0, 
    local_handle=local_handle@entry=0x7fffcd23a7b0, 
    local_offset=local_offset@entry=0, size=size@entry=64, 
    op_id=op_id@entry=0x7fffd81e0838)
    at /home/mschaara/source/deps_daos/daos_m/_build.external/mercury/src/mercury_bulk.c:1721
#7  0x00007ffff558fde2 in HG_Bulk_transfer (context=<optimized out>, 
    callback=callback@entry=0x7ffff7754080 <crt_hg_bulk_transfer_cb>, 
    arg=arg@entry=0x7fffcd23eaf0, op=<optimized out>, 
    origin_addr=<optimized out>, origin_handle=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    origin_offset=0, local_handle=0x7fffcd23a7b0, local_offset=0, size=64, 
    op_id=op_id@entry=0x7fffd81e0838)
    at /home/mschaara/source/deps_daos/daos_m/_build.external/mercury/src/mercury_bulk.c:1643
#8  0x00007ffff77591ba in crt_hg_bulk_transfer (
    bulk_desc=bulk_desc@entry=0x7fffd81e0880, 
    complete_cb=0x7fffda5c0e10 <bulk_cb>, arg=0x7fffd81e0840, 
    opid=0x7fffd81e0838) at src/cart/crt_hg.c:1654
#9  0x00007ffff77383a3 in crt_bulk_transfer (
    bulk_desc=bulk_desc@entry=0x7fffd81e0880, 
    complete_cb=complete_cb@entry=0x7fffda5c0e10 <bulk_cb>, 
    arg=arg@entry=0x7fffd81e0840, opid=opid@entry=0x7fffd81e0838)
    at src/cart/crt_bulk.c:172
#10 0x00007fffda5c0d64 in transfer_map_buf (tx=tx@entry=0x7fffd81e0a10, 
    svc=<optimized out>, rpc=rpc@entry=0x7fffcd23e348, 
    remote_bulk=0x7fffcd239840, 
    required_buf_size=required_buf_size@entry=0x7fffcd23e48c)
    at src/pool/srv_pool.c:1689
#11 0x00007fffda5c433e in ds_pool_connect_handler (rpc=0x7fffcd23e348)
    at src/pool/srv_pool.c:1801
#12 0x00007ffff7773a00 in crt_handle_rpc (arg=0x7fffcd23e348)
    at src/cart/crt_rpc.c:1745
#13 0x00007ffff6a8f708 in ABTD_thread_func_wrapper ()
---Type <return> to continue, or q <return> to quit---
   from /scratch/mschaara/DEPS//argobots/lib/libabt.so.0
#14 0x00007ffff6a8fc91 in make_fcontext ()
   from /scratch/mschaara/DEPS//argobots/lib/libabt.so.0
#15 0x0000000000000000 in ?? ()
 
Thanks,
Mohamad


----------------------------------------------------

I also realized it’s not always the same trace where it segfaults.. here is another one:
 
(gdb) bt
#0  0x00007ffff33b60ab in __psm2_mq_isend2 (mq=0x71c8d0, dest=0x7fffcd53c370, 
    flags=0, stag=0x7fffd8195ae0, buf=0x7fffd8d6d800, len=184, 
    context=0x7fffcc068788, req=0x7fffd8195ad8)
    at /scratch/ESSIO/OPA/opa-psm2/psm_mq.c:733
---Type <return> to continue, or q <return> to quit---
#1  0x00007ffff3f05594 in psmx2_tagged_send_no_flag_av_table ()
   from /scratch/mschaara/DEPS/ofi/lib/libfabric.so.1
#2  0x00007ffff536fa03 in fi_tsend (context=0x7fffcc068788, tag=4294967297, 
    dest_addr=3940649673949431, desc=0x7fffcc031a40, len=184, 
    buf=0x7fffd8d6d800, ep=0x7fde60)
    at /scratch/mschaara/DEPS/ofi/include/rdma/fi_tagged.h:116
#3  na_ofi_msg_send_expected (na_class=0x6d88d0, context=0x7fffcc02d510, 
    callback=<optimized out>, arg=<optimized out>, buf=0x7fffd8d6d800, 
    buf_size=184, plugin_data=0x7fffcc031a40, dest=0x7fffcd6316b0, 
    target_id=0 '\000', tag=1, op_id=0x7fffcc068610)
    at /home/mschaara/source/deps_daos/daos_m/_build.external/mercury/src/na/na_ofi.c:3298
#4  0x00007ffff5585583 in hg_core_respond_na (hg_core_handle=<optimized out>)
    at /home/mschaara/source/deps_daos/daos_m/_build.external/mercury/src/mercury_core.c:2084
#5  0x00007ffff558895b in HG_Core_respond (handle=0x7fffcc068510, 
    callback=callback@entry=0x7ffff558a290 <hg_core_respond_cb>, 
    arg=arg@entry=0x7fffcc068930, flags=<optimized out>, 
    payload_size=<optimized out>)
    at /home/mschaara/source/deps_daos/daos_m/_build.external/mercury/src/mercury_core.c:4794
#6  0x00007ffff558d82a in HG_Respond (handle=0x7fffcc068930, 
    callback=callback@entry=0x7ffff77547a0 <crt_hg_reply_send_cb>, 
---Type <return> to continue, or q <return> to quit---
    arg=arg@entry=0x7fffcd7280f0, out_struct=out_struct@entry=0x7fffcd728170)
    at /home/mschaara/source/deps_daos/daos_m/_build.external/mercury/src/mercury.c:2305
#7  0x00007ffff77579f1 in crt_hg_reply_send (
    rpc_priv=rpc_priv@entry=0x7fffcd7280f0) at src/cart/crt_hg.c:1312
#8  0x00007ffff77741b4 in crt_reply_send (req=req@entry=0x7fffcd728148)
    at src/cart/crt_rpc.c:1527
#9  0x00007fffda1999cf in ds_obj_rw_complete (map_version=1, status=0, 
    ioh=..., cont_hdl=<optimized out>, rpc=0x7fffcd728148)
    at src/object/srv_obj.c:102
#10 ds_obj_rw_handler (rpc=0x7fffcd728148) at src/object/srv_obj.c:724
#11 0x00007ffff7773a00 in crt_handle_rpc (arg=0x7fffcd728148)
    at src/cart/crt_rpc.c:1745
#12 0x00007ffff6a8f708 in ABTD_thread_func_wrapper ()
   from /scratch/mschaara/DEPS//argobots/lib/libabt.so.0
#13 0x00007ffff6a8fc91 in make_fcontext ()
   from /scratch/mschaara/DEPS//argobots/lib/libabt.so.0
#14 0x0000000000000000 in ?? ()

[sdwillso@boro-59 test]$ ./run_daos_tests daos:test_file
**** Testing I/O functions ****
**** Testing simple.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing async.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing async-multiple.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing coll_test.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing excl.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
../../../../src/mpi/romio/adio/ad_daos/ad_daos_open.c:281 ADIOI_DAOS_Open() - Array exists (EXCL mode) (-1004)

 No Errors
**** Testing file_info.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing i_noncontig.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing noncontig.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing noncontig_coll.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing noncontig_coll2.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing aggregation1.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing aggregation2.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing hindexed ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
-------------------------------------------------------
   [ 0 1 2 3 4 5 6 7 8 9  0 1 2 3 4 5 6 7 8 9 ]

[ 0] 0 1 2     3 4 5      D E F     G H I    
[ 1]                                         
[ 2] 6 7 8     9 : ;      J K L     M N O    
[ 3]                                         
[ 4]                                         
[ 5] X Y Z     [ \ ]      l m n     o p q    
[ 6]                                         
[ 7] ^ _ `     a b c      r s t     u v w    
[ 8]                                         
[ 9]                                         

[10] 0 1 2     3 4 5      D E F     G H I    
[11]                                         
[12] 6 7 8     9 : ;      J K L     M N O    
[13]                                         
[14]                                         
[15] X Y Z     [ \ ]      l m n     o p q    
[16]                                         
[17] ^ _ `     a b c      r s t     u v w    
[18]                                         
[19]                                         

 No Errors
**** Testing split_coll.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing psimple.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing error.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing status.c ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing types_with_zeros ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing darray_read ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
 No Errors
**** Testing fcoll_test.f ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
  No Errors
**** Testing pfcoll_test.f ****
POOL UUID = b6bc255f-c7df-4807-bb9c-44204137b1ac
SVC LIST = 0
  No Errors

DAOS Community

5-15-18