Tip of master, commit 736edae79d638e22a0601e5ad47aa95733f9d614

All tests run with ofi+psm2, ib0.

daos_test: Run with 8 server (boro-[4-11]), 2 client (boro-[12-13]). Killed servers, cleaned /mnt/daos in between runs listed below.

Tests requiring pool to be created via dmg used 4GB pool. These used boro-12 as client.

mpich tests used boro-4 as server, boro-12 as client, with a 1GB pool.

Test Results

daos_test

Separate runs with cleanup in between:

-mpcCAeoRd - PASS
-i - FAIL, still rebuilding on IO27 after 10 minutes
- DAOS-1289 - daos_test -i subtest 27 rebuild hangs OPEN
-r - FAIL
- looks to be same as -i above, still rebuilding after 10 min
-O - PASS

daosperf

1K Records

CREDITS=1

[sdwillso@boro-4 ~]$ orterun --mca mtl ^psm2,ofi -N 1 -quiet --hostfile ~/scripts/host.cli.1 --ompi-server file:~/scripts/uri.txt -x DD_SUBSYS= -x DD_MASK= -x D_LOG_FILE=/tmp/daos_perf.log daos_perf -T daos -P 2G -d 1 -a 200 -r 1000 -s 1K -C 1 -t -z
Test :
	DAOS (full stack)
Parameters :
	pool size     : 2048 MB
	credits       : 1 (sync I/O for -ve)
	obj_per_cont  : 1 x 2 (procs)
	dkey_per_obj  : 1
	akey_per_dkey : 200
	recx_per_akey : 1000
	value type    : single
	value size    : 1024
	zero copy     : yes
	overwrite     : yes
	verify fetch  : no
	VOS file      : <NULL>
151265ce: rank 1 became pool service leader 0
Started...
update successfully completed:
	duration : 91.945923  sec
	bandwith : 4.248      MB/sec
	rate     : 4350.38    IO/sec
	latency  : 229.865    us (nonsense if credits > 1)
Duration across processes:
	MAX duration : 91.945923  sec
	MIN duration : 88.257693  sec
	Average duration : 90.101808  sec
151265ce: rank 1 no longer pool service leader 0

CREDITS=8

hitting segfault
- CART-496 - Getting issue details... STATUS

4K Records

CREDITS=1

hitting segfault
- CART-496 - Getting issue details... STATUS

IOR, 10GB pool, data verification enabled

daos_bench

kv-idx-update

At end of this test with multiple servers, container destroy fails
- DAOS-1243 - Getting issue details... STATUS

Time: 580.999753 seconds (1721.171127 ops per second)

[sdwillso@boro-4 ~]$ orterun -np 1 --mca mtl ^psm2,ofi  --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-idx-update --testid=1 --svc=1 --dpool=c8ba8257-0b50-47c5-8d4f-bc08ebc1e907 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Wed Sep  5 21:50:45 2018
=================================
===============================
Test Setup
---------------
Test: kv-idx-update
DAOS pool :c8ba8257-0b50-47c5-8d4f-bc08ebc1e907
DAOS container :dd26496c-2bd3-4583-81eb-a0b826441022
Value buffer size: 64
Number of processes: 1
Number of indexes/process: 1000000
Number of asynchronous I/O: 32
===============================
kv-idx-update
Time: 580.999753 seconds (1721.171127 ops per second)

kv-dkey-update

Time: 0.217102 seconds (921.226519 ops per second)

[sdwillso@boro-4 ~]$ orterun -N 1 --mca mtl ^psm2,ofi  --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-dkey-update --testid=1 --svc=1 --dpool=b29360f3-1c1e-4af9-b6e2-f0bcf20a8a68 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Wed Sep  5 22:59:38 2018
=================================
===============================
Test Setup
---------------
Test: kv-dkey-update
DAOS pool :b29360f3-1c1e-4af9-b6e2-f0bcf20a8a68
DAOS container :b8dc329a-f72b-45b4-90d1-90ecdf16a787
Value buffer size: 64
Number of processes: 2
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-dkey-update
Time: 0.217102 seconds (921.226519 ops per second)

kv-akey-update

Time: 0.204167 seconds (979.587847 ops per second)

[sdwillso@boro-4 ~]$ orterun -N 1 --mca mtl ^psm2,ofi  --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-akey-update --testid=1 --svc=1 --dpool=bf09969d-73b0-43b4-bd75-4ad089b41849 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Wed Sep  5 23:04:18 2018
=================================
===============================
Test Setup
---------------
Test: kv-akey-update
DAOS pool :bf09969d-73b0-43b4-bd75-4ad089b41849
DAOS container :3fba85b7-3b57-4fd9-82af-4dc0821dd53a
Value buffer size: 64
Number of processes: 2
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-akey-update
Time: 0.204167 seconds (979.587847 ops per second)

kv-dkey-fetch

Time: 0.183225 seconds (1091.551968 ops per second)

[sdwillso@boro-4 ~]$ orterun -N 1 --mca mtl ^psm2,ofi  --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-dkey-fetch --testid=1 --svc=1 --dpool=528ea4d1-4a7a-4206-9ade-84c0489d928f --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Wed Sep  5 23:06:11 2018
=================================
===============================
Test Setup
---------------
Test: kv-dkey-fetch
DAOS pool :528ea4d1-4a7a-4206-9ade-84c0489d928f
DAOS container :7d77b626-b620-414a-bc05-538597fb64a2
Value buffer size: 64
Number of processes: 2
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-dkey-fetch
Time: 0.183225 seconds (1091.551968 ops per second)

kv-akey-fetch

Time: 0.146468 seconds (1365.487761 ops per second)

[sdwillso@boro-4 ~]$ orterun -N 1 --mca mtl ^psm2,ofi  --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-akey-fetch --testid=1 --svc=1 --dpool=af6b05aa-ae57-43ee-8552-f89dd7f6f271 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Wed Sep  5 23:08:02 2018
=================================
===============================
Test Setup
---------------
Test: kv-akey-fetch
DAOS pool :af6b05aa-ae57-43ee-8552-f89dd7f6f271
DAOS container :c684d271-6870-4f68-8ba5-ae3920a528e6
Value buffer size: 64
Number of processes: 2
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-akey-fetch
Time: 0.146468 seconds (1365.487761 ops per second)

CaRT Self-Test

Small IO

[sdwillso@boro-4 ~]$ orterun -np 1 -ompi-server file:~/scripts/uri.txt self_test --group-name daos_server --endpoint 0:0 --message-sizes 0 --max-inflight-rpcs 16 --repetitions 100000
Adding endpoints:
  ranks: 0 (# ranks = 1)
  tags: 0 (# tags = 1)
Warning: No --master-endpoint specified; using this command line application as the master endpoint
Self Test Parameters:
  Group name to test against: daos_server
  # endpoints:                1
  Message sizes:              [(0-EMPTY 0-EMPTY)]
  Buffer addresses end with:  <Default>
  Repetitions per size:       100000
  Max inflight RPCs:          16

host boro-4.boro.hpdd.intel.com finished self_test duration 18.610635 S.
##################################################
Results for message size (0-EMPTY 0-EMPTY) (max_inflight_rpcs = 16):

Master Endpoint 0:0
-------------------
	RPC Bandwidth (MB/sec): 0.00
	RPC Throughput (RPCs/sec): 5373
	RPC Latencies (us):
		Min    : 1054
		25th  %: 2901
		Median : 2931
		75th  %: 2958
		Max    : 13237
		Average: 2947
		Std Dev: 326.50
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 2931

Large IO Bulk PUT

[sdwillso@boro-4 ~]$ orterun -np 1 -ompi-server file:~/scripts/uri.txt self_test --group-name daos_server --endpoint 0:0 --message-sizes "0 b1048576" --max-inflight-rpcs 16 --repetitions 1000
Adding endpoints:
  ranks: 0 (# ranks = 1)
  tags: 0 (# tags = 1)
Warning: No --master-endpoint specified; using this command line application as the master endpoint
Self Test Parameters:
  Group name to test against: daos_server
  # endpoints:                1
  Message sizes:              [(0-EMPTY 1048576-BULK_PUT)]
  Buffer addresses end with:  <Default>
  Repetitions per size:       1000
  Max inflight RPCs:          16

host boro-4.boro.hpdd.intel.com finished self_test duration 0.337048 S.
##################################################
Results for message size (0-EMPTY 1048576-BULK_PUT) (max_inflight_rpcs = 16):

Master Endpoint 0:0
-------------------
	RPC Bandwidth (MB/sec): 2966.94
	RPC Throughput (RPCs/sec): 2967
	RPC Latencies (us):
		Min    : 2272
		25th  %: 5311
		Median : 5340
		75th  %: 5365
		Max    : 6216
		Average: 5325
		Std Dev: 249.22
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 5340

Large IO Bulk GET

etitions 1000
Adding endpoints:
  ranks: 0 (# ranks = 1)
  tags: 0 (# tags = 1)
Warning: No --master-endpoint specified; using this command line application as the master endpoint
Self Test Parameters:
  Group name to test against: daos_server
  # endpoints:                1
  Message sizes:              [(1048576-BULK_GET 1048576-BULK_PUT)]
  Buffer addresses end with:  <Default>
  Repetitions per size:       1000
  Max inflight RPCs:          16

host boro-4.boro.hpdd.intel.com finished self_test duration 0.585722 S.
##################################################
Results for message size (1048576-BULK_GET 1048576-BULK_PUT) (max_inflight_rpcs = 16):

Master Endpoint 0:0
-------------------
	RPC Bandwidth (MB/sec): 3414.59
	RPC Throughput (RPCs/sec): 1707
	RPC Latencies (us):
		Min    : 2926
		25th  %: 8731
		Median : 8787
		75th  %: 9713
		Max    : 14597
		Average: 9290
		Std Dev: 1592.07
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 8787

mpich tests

Results: Hangs at first test, this is known issue, CART-496 - Getting issue details... STATUS

DAOS Community

9-4-18

Test Results

daos_test

daosperf

1K Records

CREDITS=1

CREDITS=8

4K Records

CREDITS=1

IOR, 10GB pool, data verification enabled

daos_bench

kv-idx-update

kv-dkey-update

kv-akey-update

kv-dkey-fetch

kv-akey-fetch

CaRT Self-Test

Small IO

Large IO Bulk PUT

Large IO Bulk GET

mpich tests