9-26-18

Owned by Stephen Willson (Unlicensed)
Last updated: Aug 26, 2021 by Jelon Anderson (Deactivated)Version comment
8 min read

Tip of master, commit 267fa2c1343f5f72a36d5a673c7c880143fee0fe

All tests run with ofi+psm2, ib0.

daos_test: Run with 8 server (boro-[4-11]), 2 client (boro-[12-13]). Killed servers, cleaned /mnt/daos in between runs listed below.

Tests requiring pool to be created via dmg used 4GB pool. These used boro-12 as client.

mpich tests used boro-4 as server, boro-12 as client, with a 1GB pool.

Test Results

daos_test

Separate runs with cleanup in between:

  • -mpcCAeoRd - PASS
  • -i - FAIL, still rebuilding on IO27 after 10 minutes
  • -r - same as -i
  • -O - PASS

daosperf

1K Records

CREDITS=1

[sdwillso@boro-4 ~]$ orterun --mca mtl ^psm2,ofi -np 1 -quiet --hostfile ~/scripts/host.cli.1 --ompi-server file:~/scripts/uri.txt -x DD_SUBSYS= -x DD_MASK= -x D_LOG_FILE=/tmp/daos_perf.log daos_perf -T daos -P 2G -d 1 -a 200 -r 1000 -s 1K -C 1 -t -z
Test :
	DAOS (full stack)
Parameters :
	pool size     : 2048 MB
	credits       : 1 (sync I/O for -ve)
	obj_per_cont  : 1 x 1 (procs)
	dkey_per_obj  : 1
	akey_per_dkey : 200
	recx_per_akey : 1000
	value type    : single
	value size    : 1024
	zero copy     : yes
	overwrite     : yes
	verify fetch  : no
	VOS file      : <NULL>
884d80cb: rank 1 became pool service leader 0
Started...
update successfully completed:
	duration : 95.735795  sec
	bandwith : 2.040      MB/sec
	rate     : 2089.08    IO/sec
	latency  : 478.679    us (nonsense if credits > 1)
Duration across processes:
	MAX duration : 95.735795  sec
	MIN duration : 95.735795  sec
	Average duration : 95.735795  sec
884d80cb: rank 1 no longer pool service leader 0

CREDITS=8

  • CART-496 - Getting issue details... STATUS
  • Bug is fixed in patch that's not yet merged to master

4K Records

CREDITS=1

  • CART-496 - Getting issue details... STATUS
  • Bug is fixed in patch that's not yet merged to master

IOR, 50GB pool, data verification enabled

[sdwillso@boro-4 ~]$ orterun -x FI_PSM2_DISCONNECT=1 -N 1 --hostfile ~/hostlists/daos_client_hostlist --mca mtl ^psm2,ofi  --ompi-server file:~/scripts/uri.txt ior -v -W -i 1 -a DAOS -w -o `uuidgen` -b 5g -t 1m -- -p 0a410b8a-327c-4c71-8ba7-4230f390cd7d -v 1 -r 1m -s 1m -c 1024 -a 16 -o LARGE -e 1
ior WARNING: assuming POSIX-based backend for DAOS statfs call.
ior WARNING: assuming POSIX-based backend for DAOS mkdir call.
ior WARNING: assuming POSIX-based backend for DAOS rmdir call.
ior WARNING: assuming POSIX-based backend for DAOS access call.
ior WARNING: assuming POSIX-based backend for DAOS stat call.
ior WARNING: assuming POSIX-based backend for DAOS statfs call.
ior WARNING: assuming POSIX-based backend for DAOS mkdir call.
ior WARNING: assuming POSIX-based backend for DAOS rmdir call.
ior WARNING: assuming POSIX-based backend for DAOS access call.
ior WARNING: assuming POSIX-based backend for DAOS stat call.
IOR-3.1.0: MPI Coordinated Test of Parallel I/O
Began               : Thu Sep 27 22:08:13 2018
Command line        : ior -v -W -i 1 -a DAOS -w -o 4d47861b-127a-4693-8640-daa713492938 -b 5g -t 1m -- -p 0a410b8a-327c-4c71-8ba7-4230f390cd7d -v 1 -r 1m -s 1m -c 1024 -a 16 -o LARGE -e 1
Machine             : Linux boro-12.boro.hpdd.intel.com
Start time skew across all tasks: 14690266.15 sec
TestID              : 0
StartTime           : Thu Sep 27 22:08:13 2018
Path                : /home/sdwillso
FS                  : 3.8 TiB   Used FS: 13.5%   Inodes: 250.0 Mi   Used Inodes: 3.0%
Participating tasks: 2
[0] WARNING: USING daosStripeMax CAUSES READS TO RETURN INVALID DATA

Options: 
api                 : DAOS
apiVersion          : DAOS
test filename       : 4d47861b-127a-4693-8640-daa713492938
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : no tasks offsets
tasks               : 2
clients per node    : 1
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 5 GiB
aggregate filesize  : 10 GiB

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Thu Sep 27 22:08:14 2018
write     4484       5242880    1024.00    0.061408   2.20       0.027056   2.28       0   
Verifying contents of the file(s) just written.
Thu Sep 27 22:08:16 2018

remove    -          -          -          -          -          -          0.000067   0   
Max Write: 4483.50 MiB/sec (4701.29 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        4483.50    4483.50    4483.50       0.00    4483.50    4483.50    4483.50       0.00    2.28393     0      2   1    1   0     0        1         0    0      1 5368709120  1048576   10240.0 DAOS      0
Finished            : Thu Sep 27 22:08:25 2018

daos_bench

kv-idx-update

kv-dkey-update

Time: 0.111761 seconds (894.766019 ops per second)
[sdwillso@boro-4 ~]$ orterun -np 1 --mca mtl ^psm2,ofi  --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-dkey-update --testid=1 --svc=1 --dpool=14d8f2ac-1674-4b7a-9098-7d08ec97b765 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Thu Sep 27 22:36:22 2018
=================================
===============================
Test Setup
---------------
Test: kv-dkey-update
DAOS pool :14d8f2ac-1674-4b7a-9098-7d08ec97b765
DAOS container :aaccf0b3-c2c2-4c47-a047-6d2f086db318
Value buffer size: 64
Number of processes: 1
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-dkey-update
Time: 0.111761 seconds (894.766019 ops per second)

Ended at Thu Sep 27 22:36:24 2018

kv-akey-update

Time: 0.069325 seconds (1442.472191 ops per second)
[sdwillso@boro-4 ~]$ orterun -np 1 --mca mtl ^psm2,ofi  --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-akey-update --testid=1 --svc=1 --dpool=007ae409-5ee1-48d7-8dd6-4512320b52df --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Thu Sep 27 22:48:57 2018
=================================
===============================
Test Setup
---------------
Test: kv-akey-update
DAOS pool :007ae409-5ee1-48d7-8dd6-4512320b52df
DAOS container :e6c187ad-e8d7-4529-94d4-c357a1c04aee
Value buffer size: 64
Number of processes: 1
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-akey-update
Time: 0.069325 seconds (1442.472191 ops per second)

Ended at Thu Sep 27 22:48:59 2018

kv-dkey-fetch

Time: 0.073648 seconds (1357.817264 ops per second)
[sdwillso@boro-4 ~]$ orterun -np 1 --mca mtl ^psm2,ofi  --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-dkey-fetch --testid=1 --svc=1 --dpool=bd2bd088-d733-4396-963d-6d111815d4af --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000

================================
DAOSBENCH (KV)
Started at
Thu Sep 27 22:53:30 2018
=================================
===============================
Test Setup
---------------
Test: kv-dkey-fetch
DAOS pool :bd2bd088-d733-4396-963d-6d111815d4af
DAOS container :31748073-6e16-45ba-8055-568ba5cacec2
Value buffer size: 64
Number of processes: 1
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-dkey-fetch
Time: 0.073648 seconds (1357.817264 ops per second)

Ended at Thu Sep 27 22:53:31 2018

kv-akey-fetch

Time: 0.041221 seconds (2425.952090 ops per second)
[sdwillso@boro-4 ~]$ orterun -np 1 --mca mtl ^psm2,ofi  --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-akey-fetch --testid=1 --svc=1 --dpool=b294bab3-64b9-4ad5-a69a-649de9435a8f --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Thu Sep 27 22:55:49 2018
=================================
===============================
Test Setup
---------------
Test: kv-akey-fetch
DAOS pool :b294bab3-64b9-4ad5-a69a-649de9435a8f
DAOS container :5a3543e3-b34d-4651-aafa-84f9a8188d22
Value buffer size: 64
Number of processes: 1
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-akey-fetch
Time: 0.041221 seconds (2425.952090 ops per second)

Ended at Thu Sep 27 22:55:51 2018

CaRT Self-Test

Small IO

[sdwillso@boro-4 ~]$ orterun -np 1 -ompi-server file:~/scripts/uri.txt self_test --group-name daos_server --endpoint 0:0 --message-sizes 0 --max-inflight-rpcs 16 --repetitions 100000
Adding endpoints:
  ranks: 0 (# ranks = 1)
  tags: 0 (# tags = 1)
Warning: No --master-endpoint specified; using this command line application as the master endpoint
Self Test Parameters:
  Group name to test against: daos_server
  # endpoints:                1
  Message sizes:              [(0-EMPTY 0-EMPTY)]
  Buffer addresses end with:  <Default>
  Repetitions per size:       100000
  Max inflight RPCs:          16

host boro-4.boro.hpdd.intel.com finished self_test duration 19.491496 S.
##################################################
Results for message size (0-EMPTY 0-EMPTY) (max_inflight_rpcs = 16):

Master Endpoint 0:0
-------------------
	RPC Bandwidth (MB/sec): 0.00
	RPC Throughput (RPCs/sec): 5130
	RPC Latencies (us):
		Min    : 1130
		25th  %: 3054
		Median : 3076
		75th  %: 3109
		Max    : 12992
		Average: 3086
		Std Dev: 89.24
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 3076

Large IO Bulk PUT

[sdwillso@boro-4 ~]$ orterun -np 1 -ompi-server file:~/scripts/uri.txt self_test --group-name daos_server --endpoint 0:0 --message-sizes "0 b1048576" --max-inflight-rpcs 16 --repetitions 1000
Adding endpoints:
  ranks: 0 (# ranks = 1)
  tags: 0 (# tags = 1)
Warning: No --master-endpoint specified; using this command line application as the master endpoint
Self Test Parameters:
  Group name to test against: daos_server
  # endpoints:                1
  Message sizes:              [(0-EMPTY 1048576-BULK_PUT)]
  Buffer addresses end with:  <Default>
  Repetitions per size:       1000
  Max inflight RPCs:          16

host boro-4.boro.hpdd.intel.com finished self_test duration 0.337017 S.
##################################################
Results for message size (0-EMPTY 1048576-BULK_PUT) (max_inflight_rpcs = 16):

Master Endpoint 0:0
-------------------
	RPC Bandwidth (MB/sec): 2967.21
	RPC Throughput (RPCs/sec): 2967
	RPC Latencies (us):
		Min    : 2246
		25th  %: 5317
		Median : 5341
		75th  %: 5364
		Max    : 6259
		Average: 5325
		Std Dev: 254.84
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 5341

Large IO Bulk GET

[sdwillso@boro-4 ~]$ orterun -np 1 -ompi-server file:~/scripts/uri.txt self_test --group-name daos_server --endpoint 0:0 --message-sizes “b1048576 0” --max-inflight-rpcs 16 --repetitions 1000
Adding endpoints:
  ranks: 0 (# ranks = 1)
  tags: 0 (# tags = 1)
Warning: No --master-endpoint specified; using this command line application as the master endpoint
Self Test Parameters:
  Group name to test against: daos_server
  # endpoints:                1
  Message sizes:              [(1048576-BULK_GET 1048576-BULK_PUT)]
  Buffer addresses end with:  <Default>
  Repetitions per size:       1000
  Max inflight RPCs:          16

host boro-4.boro.hpdd.intel.com finished self_test duration 0.573731 S.
##################################################
Results for message size (1048576-BULK_GET 1048576-BULK_PUT) (max_inflight_rpcs = 16):

Master Endpoint 0:0
-------------------
	RPC Bandwidth (MB/sec): 3485.95
	RPC Throughput (RPCs/sec): 1743
	RPC Latencies (us):
		Min    : 4224
		25th  %: 9002
		Median : 9371
		75th  %: 9448
		Max    : 10854
		Average: 9098
		Std Dev: 722.59
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 9371

mpich tests