Tip of master, commit 736edae79d638e22a0601e5ad47aa95733f9d614
All tests run with ofi+psm2, ib0.
daos_test: Run with 8 server (boro-[4-11]), 2 client (boro-[12-13]). Killed servers, cleaned /mnt/daos in between runs listed below.
Tests requiring pool to be created via dmg used 4GB pool. These used boro-12 as client.
mpich tests used boro-4 as server, boro-12 as client, with a 1GB pool.
Test Results
daos_test
Separate runs with cleanup in between:
- -mpcCAeoRd - PASS
- -i - FAIL, still rebuilding on IO27 after 10 minutes
DAOS-1289 - daos_test -i subtest 27 rebuild hangs OPEN
- -r - FAIL
- looks to be same as -i above, still rebuilding after 10 min
- -O - PASS
daosperf
1K Records
CREDITS=1
[sdwillso@boro-4 ~]$ orterun --mca mtl ^psm2,ofi -N 1 -quiet --hostfile ~/scripts/host.cli.1 --ompi-server file:~/scripts/uri.txt -x DD_SUBSYS= -x DD_MASK= -x D_LOG_FILE=/tmp/daos_perf.log daos_perf -T daos -P 2G -d 1 -a 200 -r 1000 -s 1K -C 1 -t -z
Test :
DAOS (full stack)
Parameters :
pool size : 2048 MB
credits : 1 (sync I/O for -ve)
obj_per_cont : 1 x 2 (procs)
dkey_per_obj : 1
akey_per_dkey : 200
recx_per_akey : 1000
value type : single
value size : 1024
zero copy : yes
overwrite : yes
verify fetch : no
VOS file : <NULL>
151265ce: rank 1 became pool service leader 0
Started...
update successfully completed:
duration : 91.945923 sec
bandwith : 4.248 MB/sec
rate : 4350.38 IO/sec
latency : 229.865 us (nonsense if credits > 1)
Duration across processes:
MAX duration : 91.945923 sec
MIN duration : 88.257693 sec
Average duration : 90.101808 sec
151265ce: rank 1 no longer pool service leader 0
CREDITS=8
4K Records
CREDITS=1
- hitting segfault
-
Unable to locate Jira server for this macro. It may be due to Application Link configuration.
IOR, 10GB pool, data verification enabled
daos_bench
kv-idx-update
- At end of this test with multiple servers, container destroy fails
-
Unable to locate Jira server for this macro. It may be due to Application Link configuration.
Time: 580.999753 seconds (1721.171127 ops per second)
[sdwillso@boro-4 ~]$ orterun -np 1 --mca mtl ^psm2,ofi --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-idx-update --testid=1 --svc=1 --dpool=c8ba8257-0b50-47c5-8d4f-bc08ebc1e907 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Wed Sep 5 21:50:45 2018
=================================
===============================
Test Setup
---------------
Test: kv-idx-update
DAOS pool :c8ba8257-0b50-47c5-8d4f-bc08ebc1e907
DAOS container :dd26496c-2bd3-4583-81eb-a0b826441022
Value buffer size: 64
Number of processes: 1
Number of indexes/process: 1000000
Number of asynchronous I/O: 32
===============================
kv-idx-update
Time: 580.999753 seconds (1721.171127 ops per second)
kv-dkey-update
Time: 0.217102 seconds (921.226519 ops per second)
[sdwillso@boro-4 ~]$ orterun -N 1 --mca mtl ^psm2,ofi --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-dkey-update --testid=1 --svc=1 --dpool=b29360f3-1c1e-4af9-b6e2-f0bcf20a8a68 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Wed Sep 5 22:59:38 2018
=================================
===============================
Test Setup
---------------
Test: kv-dkey-update
DAOS pool :b29360f3-1c1e-4af9-b6e2-f0bcf20a8a68
DAOS container :b8dc329a-f72b-45b4-90d1-90ecdf16a787
Value buffer size: 64
Number of processes: 2
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-dkey-update
Time: 0.217102 seconds (921.226519 ops per second)
kv-akey-update
Time: 0.204167 seconds (979.587847 ops per second)
[sdwillso@boro-4 ~]$ orterun -N 1 --mca mtl ^psm2,ofi --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-akey-update --testid=1 --svc=1 --dpool=bf09969d-73b0-43b4-bd75-4ad089b41849 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Wed Sep 5 23:04:18 2018
=================================
===============================
Test Setup
---------------
Test: kv-akey-update
DAOS pool :bf09969d-73b0-43b4-bd75-4ad089b41849
DAOS container :3fba85b7-3b57-4fd9-82af-4dc0821dd53a
Value buffer size: 64
Number of processes: 2
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-akey-update
Time: 0.204167 seconds (979.587847 ops per second)
kv-dkey-fetch
Time: 0.183225 seconds (1091.551968 ops per second)
[sdwillso@boro-4 ~]$ orterun -N 1 --mca mtl ^psm2,ofi --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-dkey-fetch --testid=1 --svc=1 --dpool=528ea4d1-4a7a-4206-9ade-84c0489d928f --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Wed Sep 5 23:06:11 2018
=================================
===============================
Test Setup
---------------
Test: kv-dkey-fetch
DAOS pool :528ea4d1-4a7a-4206-9ade-84c0489d928f
DAOS container :7d77b626-b620-414a-bc05-538597fb64a2
Value buffer size: 64
Number of processes: 2
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-dkey-fetch
Time: 0.183225 seconds (1091.551968 ops per second)
kv-akey-fetch
Time: 0.146468 seconds (1365.487761 ops per second)
[sdwillso@boro-4 ~]$ orterun -N 1 --mca mtl ^psm2,ofi --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-akey-fetch --testid=1 --svc=1 --dpool=af6b05aa-ae57-43ee-8552-f89dd7f6f271 --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
================================
DAOSBENCH (KV)
Started at
Wed Sep 5 23:08:02 2018
=================================
===============================
Test Setup
---------------
Test: kv-akey-fetch
DAOS pool :af6b05aa-ae57-43ee-8552-f89dd7f6f271
DAOS container :c684d271-6870-4f68-8ba5-ae3920a528e6
Value buffer size: 64
Number of processes: 2
Number of keys/process: 100
Number of asynchronous I/O: 32
===============================
kv-akey-fetch
Time: 0.146468 seconds (1365.487761 ops per second)
CaRT Self-Test
Small IO
[sdwillso@boro-4 ~]$ orterun -np 1 -ompi-server file:~/scripts/uri.txt self_test --group-name daos_server --endpoint 0:0 --message-sizes 0 --max-inflight-rpcs 16 --repetitions 100000
Adding endpoints:
ranks: 0 (# ranks = 1)
tags: 0 (# tags = 1)
Warning: No --master-endpoint specified; using this command line application as the master endpoint
Self Test Parameters:
Group name to test against: daos_server
# endpoints: 1
Message sizes: [(0-EMPTY 0-EMPTY)]
Buffer addresses end with: <Default>
Repetitions per size: 100000
Max inflight RPCs: 16
host boro-4.boro.hpdd.intel.com finished self_test duration 18.610635 S.
##################################################
Results for message size (0-EMPTY 0-EMPTY) (max_inflight_rpcs = 16):
Master Endpoint 0:0
-------------------
RPC Bandwidth (MB/sec): 0.00
RPC Throughput (RPCs/sec): 5373
RPC Latencies (us):
Min : 1054
25th %: 2901
Median : 2931
75th %: 2958
Max : 13237
Average: 2947
Std Dev: 326.50
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 2931
Large IO Bulk PUT
[sdwillso@boro-4 ~]$ orterun -np 1 -ompi-server file:~/scripts/uri.txt self_test --group-name daos_server --endpoint 0:0 --message-sizes "0 b1048576" --max-inflight-rpcs 16 --repetitions 1000
Adding endpoints:
ranks: 0 (# ranks = 1)
tags: 0 (# tags = 1)
Warning: No --master-endpoint specified; using this command line application as the master endpoint
Self Test Parameters:
Group name to test against: daos_server
# endpoints: 1
Message sizes: [(0-EMPTY 1048576-BULK_PUT)]
Buffer addresses end with: <Default>
Repetitions per size: 1000
Max inflight RPCs: 16
host boro-4.boro.hpdd.intel.com finished self_test duration 0.337048 S.
##################################################
Results for message size (0-EMPTY 1048576-BULK_PUT) (max_inflight_rpcs = 16):
Master Endpoint 0:0
-------------------
RPC Bandwidth (MB/sec): 2966.94
RPC Throughput (RPCs/sec): 2967
RPC Latencies (us):
Min : 2272
25th %: 5311
Median : 5340
75th %: 5365
Max : 6216
Average: 5325
Std Dev: 249.22
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 5340
Large IO Bulk GET
etitions 1000
Adding endpoints:
ranks: 0 (# ranks = 1)
tags: 0 (# tags = 1)
Warning: No --master-endpoint specified; using this command line application as the master endpoint
Self Test Parameters:
Group name to test against: daos_server
# endpoints: 1
Message sizes: [(1048576-BULK_GET 1048576-BULK_PUT)]
Buffer addresses end with: <Default>
Repetitions per size: 1000
Max inflight RPCs: 16
host boro-4.boro.hpdd.intel.com finished self_test duration 0.585722 S.
##################################################
Results for message size (1048576-BULK_GET 1048576-BULK_PUT) (max_inflight_rpcs = 16):
Master Endpoint 0:0
-------------------
RPC Bandwidth (MB/sec): 3414.59
RPC Throughput (RPCs/sec): 1707
RPC Latencies (us):
Min : 2926
25th %: 8731
Median : 8787
75th %: 9713
Max : 14597
Average: 9290
Std Dev: 1592.07
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 8787
mpich tests
Results: Hangs at first test, this is known issue,
Unable to locate Jira server for this macro. It may be due to Application Link configuration.