Table of Contents
Introduction
This is a quick start guide to help set up daos and use it with POSIX on Centos 7.9. This document covers any pre-requisites required before starting, pointer to setting up of daos system which consists of installation of daos rpms, setting up of various configuration files needed by daos, getting daos servers and agents up and running. Once the daos system is ready for use, this document can help set up dfuse mount point in order to take advantage of daos support for POSIX, help with some example runs of daos tests and benchmarking tools like ior and mdtest along with some examples of how to move data between a POSIX file system and daos containers (and vise versa) and finally cleaning up your daos setup.
We are using a set of 2 servers each with PMEM and SSDs connected via storage network and 2 client nodes without pmem/ssd but on the storage network. We are also using another node as admin node. This can either be a separate node or any of your client nodes as well. All nodes have a base Centos 7.9 install.
Requirements
Set environment variables for list of servers, clients and admin node.
export ADMIN_NODE=node-1 export SERVER_NODES=node-2,node-3 export CLIENT_NODES=node-4,node-5 export ALL_NODES=$ADMIN_NODE,$SERVER_NODES,$CLIENT_NODES
All nodes have Centos 7.9 installed
Set password less ssh amongst all nodes
Make sure to have sudo privileges on all nodes
Enable IOMMU
If using nvme, enable IOMMU on server nodes: https://daos-stack.github.io/admin/predeployment_check/#enable-iommu-optional
Install pdsh on admin node
# Centos sudo yum install -y pdsh
Set-Up
Please refer here for initial set up which consists of rpm installation, generate and set up certificates, setting up config files, starting servers and agents.
Run daos autotest
daos autotest is a smoke test which can be used to test the setup. It performs various activities like connecting a pool, creating and opening a container, reading and writing to the container.
# create pool dmg pool create --size=50G # sample output Creating DAOS pool with automatic storage allocation: 50 GB NVMe + 6.00% SCM Pool created with 6.00% SCM/NVMe ratio --------------------------------------- UUID : 6af46954-f704-45f0-8b80-115ef855a065 Service Ranks : [1-3] Storage Ranks : [0-3] Total Size : 53 GB SCM : 3.0 GB (750 MB / rank) NVMe : 50 GB (12 GB / rank) # assign pool uuid to a variable export DAOS_POOL=<pool uuid> # run daos autotest daos pool autotest --pool $DAOS_POOL # Sample output Step Operation Status Time(sec) Comment 0 Initializing DAOS OK 0.000 1 Connecting to pool OK 0.070 2 Creating container OK 0.000 uuid = 3 Opening container OK 0.050 10 Generating 1M S1 layouts OK 4.620 11 Generating 10K SX layouts OK 0.140 20 Inserting 1M 128B values OK 75.130 21 Reading 128B values back OK 71.540 24 Inserting 1M 4KB values OK 109.190 25 Reading 4KB values back OK 103.620 28 Inserting 100K 1MB values OK 413.730 29 Reading 1MB values back OK 461.220 96 Closing container OK 0.040 97 Destroying container OK 0.070 98 Disconnecting from pool OK 0.000 99 Tearing down DAOS OK 0.000
Run cart self_test
On the client run a cart self_test to verify basic network connectivity.
# set env SHARED_DIR=<shared dir by all nodes> export FI_UNIVERSE_SIZE=2048 export OFI_INTERFACE=ib0 # generate the attach info file daos_agent -o /etc/daos/daos_agent.yml -l $SHARED_DIR/daos_agent.log dump-attachinfo -o $SHARED_DIR/daos_server.attach_info_tmp # selt_test --help for more details on params # for 4 servers --endpoint 0-3:0-1 ranks:tags. self_test --path $SHARED_DIR --group-name daos_server --endpoint 0-1:0-1 Adding endpoints: ranks: 0-1 (# ranks = 2) tags: 0-1 (# tags = 2) Warning: No --master-endpoint specified; using this command line application as the master endpoint Self Test Parameters: Group name to test against: daos_server # endpoints: 4 Message sizes: [(200000-BULK_GET 200000-BULK_PUT), (200000-BULK_GET 0-EMPTY), (0-EMPTY 200000-BULK_PUT), (200000-BULK_GET 1000-IOV), (1000-IOV 200000-BULK_PUT), (1000-IOV 1000-IOV), (1000-IOV 0-EMPTY), (0-EMPTY 1000-IOV), (0-EMPTY 0-EMPTY)] Buffer addresses end with: <Default> Repetitions per size: 40000 Max inflight RPCs: 1000 CLI [rank=0 pid=40050] Attached daos_server ################################################## Results for message size (200000-BULK_GET 200000-BULK_PUT) (max_inflight_rpcs = 1000): Master Endpoint 2:0 ------------------- RPC Bandwidth (MB/sec): 197.56 RPC Throughput (RPCs/sec): 518 RPC Latencies (us): Min : 38791 25th %: 1695365 Median : 1916632 75th %: 2144087 Max : 2969361 Average: 1907415 Std Dev: 373832.81 RPC Failures: 0 Endpoint results (rank:tag - Median Latency (us)): 0:0 - 1889518 0:1 - 1712934 1:0 - 1924995 1:1 - 2110649 ################################################## Results for message size (200000-BULK_GET 0-EMPTY) (max_inflight_rpcs = 1000): Master Endpoint 2:0 ------------------- RPC Bandwidth (MB/sec): 112.03 RPC Throughput (RPCs/sec): 587 RPC Latencies (us): Min : 4783 25th %: 1480053 Median : 1688064 75th %: 1897392 Max : 2276555 Average: 1681303 Std Dev: 314999.11 RPC Failures: 0 Endpoint results (rank:tag - Median Latency (us)): 0:0 - 2001222 0:1 - 1793990 1:0 - 1385306 1:1 - 1593675 ################################################## Results for message size (0-EMPTY 200000-BULK_PUT) (max_inflight_rpcs = 1000): Master Endpoint 2:0 ------------------- RPC Bandwidth (MB/sec): 112.12 RPC Throughput (RPCs/sec): 588 RPC Latencies (us): Min : 6302 25th %: 1063532 Median : 1654468 75th %: 2287784 Max : 3488227 Average: 1680617 Std Dev: 880402.68 RPC Failures: 0 Endpoint results (rank:tag - Median Latency (us)): 0:0 - 1251585 0:1 - 1323953 1:0 - 2099173 1:1 - 2043352 ################################################## Results for message size (200000-BULK_GET 1000-IOV) (max_inflight_rpcs = 1000): Master Endpoint 2:0 ------------------- RPC Bandwidth (MB/sec): 112.54 RPC Throughput (RPCs/sec): 587 RPC Latencies (us): Min : 5426 25th %: 1395359 Median : 1687404 75th %: 1983402 Max : 2426175 Average: 1681970 Std Dev: 393256.99 RPC Failures: 0 Endpoint results (rank:tag - Median Latency (us)): 0:0 - 2077476 0:1 - 1870102 1:0 - 1318136 1:1 - 1529193 ################################################## Results for message size (1000-IOV 200000-BULK_PUT) (max_inflight_rpcs = 1000): Master Endpoint 2:0 ------------------- RPC Bandwidth (MB/sec): 112.66 RPC Throughput (RPCs/sec): 588 RPC Latencies (us): Min : 5340 25th %: 442729 Median : 1221371 75th %: 2936906 Max : 3502405 Average: 1681142 Std Dev: 1308472.80 RPC Failures: 0 Endpoint results (rank:tag - Median Latency (us)): 0:0 - 3006315 0:1 - 2913808 1:0 - 434763 1:1 - 465469 ################################################## Results for message size (1000-IOV 1000-IOV) (max_inflight_rpcs = 1000): Master Endpoint 2:0 ------------------- RPC Bandwidth (MB/sec): 80.71 RPC Throughput (RPCs/sec): 42315 RPC Latencies (us): Min : 1187 25th %: 20187 Median : 23322 75th %: 26833 Max : 30246 Average: 23319 Std Dev: 4339.87 RPC Failures: 0 Endpoint results (rank:tag - Median Latency (us)): 0:0 - 26828 0:1 - 26839 1:0 - 20275 1:1 - 20306 ################################################## Results for message size (1000-IOV 0-EMPTY) (max_inflight_rpcs = 1000): Master Endpoint 2:0 ------------------- RPC Bandwidth (MB/sec): 42.68 RPC Throughput (RPCs/sec): 44758 RPC Latencies (us): Min : 935 25th %: 15880 Median : 21444 75th %: 28434 Max : 34551 Average: 22035 Std Dev: 7234.26 RPC Failures: 0 Endpoint results (rank:tag - Median Latency (us)): 0:0 - 28418 0:1 - 28449 1:0 - 16301 1:1 - 16318 ################################################## Results for message size (0-EMPTY 1000-IOV) (max_inflight_rpcs = 1000): Master Endpoint 2:0 ------------------- RPC Bandwidth (MB/sec): 42.91 RPC Throughput (RPCs/sec): 44991 RPC Latencies (us): Min : 789 25th %: 20224 Median : 22195 75th %: 24001 Max : 26270 Average: 21943 Std Dev: 3039.50 RPC Failures: 0 Endpoint results (rank:tag - Median Latency (us)): 0:0 - 24017 0:1 - 23987 1:0 - 20279 1:1 - 20309 ################################################## Results for message size (0-EMPTY 0-EMPTY) (max_inflight_rpcs = 1000): Master Endpoint 2:0 ------------------- RPC Bandwidth (MB/sec): 0.00 RPC Throughput (RPCs/sec): 47807 RPC Latencies (us): Min : 774 25th %: 16161 Median : 20419 75th %: 25102 Max : 29799 Average: 20633 Std Dev: 5401.96 RPC Failures: 0 Endpoint results (rank:tag - Median Latency (us)): 0:0 - 25103 0:1 - 25099 1:0 - 16401 1:1 - 16421
Run daos_perf
On the client run daos_perf. Requires openmpi3.
(No need to create pool. Daos_perf will take care of it.)
Load mpi or set it in your environment.
# load openmpi module or set it's path in your environment module load mpi/openmpi3-x86_64 or export LD_LIBRARY_PATH=<openmpi lib path>:$LD_LIBRARY_PATH export PATH=<openmpi bin path>:$PATH
Single Process daos_perf run example:
export D_LOG_FILE=/tmp/daos_perf_daos.log daos_perf -g /etc/daos/daos.yml -o 10 -d 10 -a 10 -C 16 -A -R 'U;p F;i=5;p V' -P 1G -N 10G 'U' means UPDATE, 'p' means outputing performance 'F' means FETCH, parameter "i=5" indicates the FETCH test will repeat 5 times. 'V' means verify correctness of data ‘o’ Number of Objects ‘d’ Number of dkeys ‘a’ Number of akeys ‘A’ Use Array value of akey. Single value is selected by default. ‘P’ Scm Pool size ‘N’ Nvme Pool Size Test : DAOS LARGE (full stack, non-replica) Pool : 79cae139-272e-4ada-922e-98cc968126bf Parameters : pool size : SCM: 1024 MB, NVMe: 10240 MB credits : 16 (sync I/O for -ve) obj_per_cont : 10 x 1 (procs) dkey_per_obj : 10 akey_per_dkey : 10 recx_per_akey : 16 value type : array stride size : 64 zero copy : no VOS file : <NULL> Running test=UPDATE Running UPDATE test (iteration=1) UPDATE successfully completed: duration : 0.225937 sec bandwith : 4.322 MB/sec rate : 70816.20 IO/sec latency : 14.121 us (nonsense if credits > 1) Duration across processes: MAX duration : 0.225937 sec MIN duration : 0.225937 sec Average duration : 0.225937 sec Completed test=UPDATE Running test=FETCH Running FETCH test (iteration=5) FETCH successfully completed: duration : 1.190614 sec bandwith : 4.101 MB/sec rate : 67192.22 IO/sec latency : 14.883 us (nonsense if credits > 1) Duration across processes: MAX duration : 1.190614 sec MIN duration : 1.190614 sec Average duration : 1.190614 sec
Multi-process daos_perf example:
/usr/lib64/openmpi3/bin/orterun --enable-recovery --hostfile /path/to/hostfile_clients --map-by node --mca btl_openib_warn_default_gid_prefix "0" --mca btl "tcp,self" --mca oob "tcp" --mca pml "ob1" --mca btl_tcp_if_include "eth0" --np 4 --tag-output daos_perf -g /etc/daos/daos.yml -o 10 -d 10 -a 10 -C 16 -A -R 'U;p F;i=5;p V' -P 1G -N 10G [1,0]<stdout>:Test : [1,0]<stdout>: DAOS LARGE (full stack, non-replica) [1,0]<stdout>:Pool : [1,0]<stdout>: 8ce8ae49-57e2-4d6e-b497-b50bbd6375dd [1,0]<stdout>:Parameters : [1,0]<stdout>: pool size : SCM: 1024 MB, NVMe: 10240 MB [1,0]<stdout>: credits : 16 (sync I/O for -ve) [1,0]<stdout>: obj_per_cont : 10 x 4 (procs) [1,0]<stdout>: dkey_per_obj : 10 [1,0]<stdout>: akey_per_dkey : 10 [1,0]<stdout>: recx_per_akey : 16 [1,0]<stdout>: value type : array [1,0]<stdout>: stride size : 64 [1,0]<stdout>: zero copy : no [1,0]<stdout>: VOS file : <NULL> [1,0]<stdout>:Running test=UPDATE [1,0]<stdout>:Running UPDATE test (iteration=1) [1,2]<stdout>:Running test=UPDATE [1,2]<stdout>:Running UPDATE test (iteration=1) [1,1]<stdout>:Running test=UPDATE [1,1]<stdout>:Running UPDATE test (iteration=1) [1,3]<stdout>:Running test=UPDATE [1,3]<stdout>:Running UPDATE test (iteration=1) [1,2]<stdout>:Completed test=UPDATE [1,2]<stdout>:Running test=FETCH [1,2]<stdout>:Running FETCH test (iteration=5) [1,0]<stdout>:UPDATE successfully completed: [1,1]<stdout>:Completed test=UPDATE [1,1]<stdout>:Running test=FETCH [1,0]<stdout>: duration : 64.994374 sec [1,0]<stdout>: bandwith : 0.060 MB/sec [1,0]<stdout>: rate : 984.70 IO/sec [1,0]<stdout>: latency : 22.106 us (nonsense if credits > 1) [1,0]<stdout>:Duration across processes: [1,0]<stdout>: MAX duration : 1.414762 sec [1,0]<stdout>: MIN duration : 1.064101 sec [1,3]<stdout>:Completed test=UPDATE [1,3]<stdout>:Running test=FETCH [1,3]<stdout>:Running FETCH test (iteration=5) [1,1]<stdout>:Running FETCH test (iteration=5) [1,0]<stdout>: Average duration : 1.194978 sec [1,0]<stdout>:Completed test=UPDATE [1,0]<stdout>:Running test=FETCH [1,0]<stdout>:Running FETCH test (iteration=5) [1,2]<stdout>:Completed test=FETCH [1,2]<stdout>:Running test=VERIFY [1,2]<stdout>:Running VERIFY test (iteration=1) [1,0]<stdout>:FETCH successfully completed: [1,0]<stdout>: duration : 68.807437 sec [1,0]<stdout>: bandwith : 0.284 MB/sec [1,0]<stdout>: rate : 4650.66 IO/sec [1,0]<stdout>: latency : 16.333 us (nonsense if credits > 1) [1,0]<stdout>:Duration across processes: [1,0]<stdout>: MAX duration : 5.226541 sec [1,0]<stdout>: MIN duration : 3.616002 sec [1,0]<stdout>: Average duration : 4.349847 sec [1,0]<stdout>:Completed test=FETCH [1,0]<stdout>:Running test=VERIFY [1,0]<stdout>:Running VERIFY test (iteration=1) [1,3]<stdout>:Completed test=FETCH [1,3]<stdout>:Running test=VERIFY [1,1]<stdout>:Completed test=FETCH [1,1]<stdout>:Running test=VERIFY [1,1]<stdout>:Running VERIFY test (iteration=1) [1,3]<stdout>:Running VERIFY test (iteration=1) [1,0]<stdout>:Completed test=VERIFY [1,2]<stdout>:Completed test=VERIFY [1,1]<stdout>:Completed test=VERIFY [1,3]<stdout>:Completed test=VERIFY
Run daos_racer
On the client run daos_racer. Requires openmpi3.
(No need to create pool. Daos_racer will take care of it.)
Steps involve loading of openmpi or setting in environment if not already done. Set log file location and log level, and finally run.
# load openmpi module or set it's path in your environment module load mpi/openmpi3-x86_64 or export LD_LIBRARY_PATH=<openmpi lib path>:$LD_LIBRARY_PATH export PATH=<openmpi bin path>:$PATH # set environment var export D_LOG_FILE=/tmp/daos_racer_log export D_LOG_MASK=ERR # run daos_racer /usr/bin/daos_racer -n /etc/daos/daos_control.yml -t 60
Run daos_test
On the client run daos_test. Requires openmpi3.
(No need to create pool. Daos_test will take care of it.)
Steps involve loading of openmpi or setting in environment if not already done. Set pool size environment variables, and finally run desired or all daos_tests.
# load openmpi module or set it's path in your environment
module load mpi/openmpi3-x86_64
or
export LD_LIBRARY_PATH=<openmpi lib path>:$LD_LIBRARY_PATH
export PATH=<openmpi bin path>:$PATH
export POOL_SCM_SIZE=8G
export POOL_NVME_SIZE=16G
(Example of Management Test)
/usr/lib64/openmpi3/bin/orterun --mca btl_openib_warn_default_gid_prefix 0 --mca pml ob1 --mca btl tcp,self --mca oob tcp --mca btl_tcp_if_include eth0 -n 1 --hostfile /path/to/hostfile_clients -x D_LOG_FILE=/tmp/daos_test.log --map-by node -x D_LOG_MASK=DEBUG -x DD_MASK=mgmt,io,md,epc,rebuild -x COVFILE=/tmp/test.cov /usr/bin/daos_test -n /etc/daos/daos_control.yml -m
=================
DAOS management tests..
=====================
[==========] Running 5 test(s).
[ RUN ] MGMT1: create/destroy pool on all tgts
creating pool synchronously ... success uuid = 6e3dbfee-ee2e-47ec-8b3c-6a42b62d7573
destroying pool synchronously ... success
[ OK ] MGMT1: create/destroy pool on all tgts
[ RUN ] MGMT2: create/destroy pool on all tgts (async)
async not supported[ OK ] MGMT2: create/destroy pool on all tgts (async)
[ RUN ] MGMT3: list-pools with no pools in sys
success t0: output npools=0
verifying pools[0..10], nfilled=0
success t1: pools[] over-sized
success t2: npools=0, non-NULL pools[] rc=0
success t3: in &npools NULL, -DER_INVAL
success
[ OK ] MGMT3: list-pools with no pools in sys
[ RUN ] MGMT4: list-pools with multiple pools in sys
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool 24aea230-2366-45a6-90e8-63ab36e58f8b
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool cb936b0c-c32f-4465-86f9-d18aa94a5338
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool 02f0d551-843d-4677-9905-696205c1fb5c
success t0: output npools=4
verifying pools[0..14], nfilled=4
pool 24aea230-2366-45a6-90e8-63ab36e58f8b found in list result
pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e found in list result
pool cb936b0c-c32f-4465-86f9-d18aa94a5338 found in list result
pool 02f0d551-843d-4677-9905-696205c1fb5c found in list result
success t1: pools[] over-sized
success t2: npools=0, non-NULL pools[] rc=-1016
success t3: in &npools NULL, -DER_INVAL
verifying pools[0..4], nfilled=4
pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e found in list result
pool cb936b0c-c32f-4465-86f9-d18aa94a5338 found in list result
pool 02f0d551-843d-4677-9905-696205c1fb5c found in list result
pool 24aea230-2366-45a6-90e8-63ab36e58f8b found in list result
success t4: pools[] exact length
verifying pools[0..3], nfilled=0
success t5: pools[] under-sized
success
teardown: destroyed pool 24aea230-2366-45a6-90e8-63ab36e58f8b
teardown: destroyed pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e
teardown: destroyed pool cb936b0c-c32f-4465-86f9-d18aa94a5338
teardown: destroyed pool 02f0d551-843d-4677-9905-696205c1fb5c
[ OK ] MGMT4: list-pools with multiple pools in sys
[ RUN ] MGMT5: retry MGMT_POOL_{CREATE,DESETROY} upon errors
setting DAOS_POOL_CREATE_FAIL_CORPC ... success
creating pool synchronously ... success uuid = a93bd58f-1c9a-4adf-8a96-e366e6f25c6a
destroying pool synchronously ... success
[ OK ] MGMT5: retry MGMT_POOL_{CREATE,DESETROY} upon errors
[==========] 5 test(s) run.
[ PASSED ] 5 test(s).
============ Summary src/tests/suite/daos_test.c
OK - NO TEST FAILURES
All DAOS TESTS ============================= Tests: Use one of these arg(s) for specific test daos_test -m|--mgmt daos_test -p|--daos_pool_tests daos_test -c|--daos_container_tests daos_test -C|--capa daos_test -U|--dedup daos_test -z|--checksum daos_test -Z|--ec_aggregation daos_test -t|--base_tx daos_test -T|--dist_tx daos_test -i|--daos_io_tests daos_test -x|--epoch_io daos_test -A|--array daos_test -D|--daos_array daos_test -K|--daos_kv daos_test -d|--degraded daos_test -e|--daos_epoch_tests daos_test -o|--daos_epoch_recovery_tests daos_test -V|--verify_consistency daos_test -R|--MD_replication_tests daos_test -O|--oid_alloc daos_test -r|--rebuild daos_test -v|--rebuild_simple daos_test -S|--rebuild_ec daos_test -X|--degrade_ec daos_test -b|--drain_simple daos_test -B|--extend_simple daos_test -N|--nvme_recovery daos_test -a|--daos_all_tests Default <daos_tests> runs all tests
Create pool, posix container and then mount the container via fuse
Steps:
Create a pool.
Create a posix type container.
Make directory for dfuse mount point.
Mount dfuse.
Verify dfuse got mounted.
# create pool dmg --config-path=/etc/daos/daos_control.yml pool create --size=800G Creating DAOS pool with automatic storage allocation: 800 GB NVMe + 6.00% SCM Pool created with 6.00% SCM/NVMe ratio --------------------------------------- UUID : b22220ea-740d-46bc-84ad-35ed3a28aa31 Service Ranks : [1-3] Storage Ranks : [0-7] Total Size : 848 GB SCM : 48 GB (6.0 GB / rank) NVMe : 800 GB (100 GB / rank) # List pool dmg --config-path=/etc/daos/daos_control.yml pool list Pool UUID Svc Replicas --------- ------------ b22220ea-740d-46bc-84ad-35ed3a28aa31 [1-3] # assign pool uuid to DAOS_POOL variable export DAOS_POOL=b22220ea-740d-46bc-84ad-35ed3a28aa31 # Pool Query dmg --config-path=/etc/daos/daos_control.yml pool query --pool $DAOS_POOL Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1 Pool space info: - Target(VOS) count:64 - SCM: Total size: 48 GB Free: 48 GB, min:750 MB, max:750 MB, mean:750 MB - NVMe: Total size: 800 GB Free: 660 GB, min:10 GB, max:10 GB, mean:10 GB Rebuild idle, 0 objs, 0 recs # create container of type POSIX daos container create --pool=$DAOS_POOL --type=POSIX Successfully created container cd46cf6e-f886-4682-8077-e3cbcd09b43a # assign container uuid to DAOS_CONT variable export DAOS_CONT=cd46cf6e-f886-4682-8077-e3cbcd09b43a # create dir for dfuse mountpoint on all client nodes pdsh -w $CLIENT_NODES 'mkdir /tmp/daos_dfuse' # mount dfuse pdsh -w $CLIENT_NODES dfuse --pool=$DAOS_POOL --cont=$DAOS_CONT --mountpoint=/tmp/daos_dfuse/ #verify dfuse got mounted. pdsh -w $CLIENT_NODES df /tmp/daos_dfuse wolf-86: Filesystem 1K-blocks Used Available Use% Mounted on wolf-86: dfuse 828125000 136841868 691283133 17% /tmp/daos_dfuse wolf-88: Filesystem 1K-blocks Used Available Use% Mounted on wolf-88: dfuse 828125000 136841868 691283133 17% /tmp/daos_dfuse wolf-87: Filesystem 1K-blocks Used Available Use% Mounted on wolf-87: dfuse 828125000 136841868 691283133 17% /tmp/daos_dfuse
Run IOR
Use IOR to generate an HPC type POSIX I/O load to the POSIX container.
Build ior
git clone https://github.com/hpc/ior.git cd ior ./bootstrap mkdir build;cd build ../configure --with-daos=/usr --prefix=<your dir> make make install
Add <your dir>/lib to LD_LIBRARY_PATh and <your dir>/bin to PATH
Or
On all client nodes:
sudo yum install ior
Use ior to write and read around 150G of data
# load mpich module or set it's path in your environment module load mpi/mpich-x86_64 or export LD_LIBRARY_PATH=<mpich lib path>:$LD_LIBRARY_PATH export PATH=<mpich bin path>:$PATH mpirun -hostfile /path/to/hostfile_clients -np 30 ior -a POSIX -b 5G -t 1M -v -W -w -r -R -i 1 -k -o /tmp/daos_dfuse/testfile IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O Began : Thu Apr 29 23:23:09 2021 Command line : ior -a POSIX -b 5G -t 1M -v -W -w -r -R -i 1 -k -o /tmp/daos_dfuse/testfile Machine : Linux wolf-86.wolf.hpdd.intel.com Start time skew across all tasks: 0.00 sec TestID : 0 StartTime : Thu Apr 29 23:23:09 2021 Path : /tmp/daos_dfuse/testfile FS : 789.8 GiB Used FS: 16.5% Inodes: -0.0 Mi Used Inodes: 0.0% Participating tasks : 30 Options: api : POSIX apiVersion : test filename : /tmp/daos_dfuse/testfile access : single-shared-file type : independent segments : 1 ordering in a file : sequential ordering inter file : no tasks offsets nodes : 3 tasks : 30 clients per node : 10 repetitions : 1 xfersize : 1 MiB blocksize : 5 GiB aggregate filesize : 150 GiB verbose : 1 Results: access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ---- Commencing write performance test: Thu Apr 29 23:23:09 2021 write 1299.23 1299.84 0.022917 5242880 1024.00 10.79 118.17 0.000377 118.22 0 Verifying contents of the file(s) just written. Thu Apr 29 23:25:07 2021 Commencing read performance test: Thu Apr 29 23:25:35 2021 read 5429 5431 0.005523 5242880 1024.00 0.012188 28.28 0.000251 28.29 0 Max Write: 1299.23 MiB/sec (1362.35 MB/sec) Max Read: 5429.38 MiB/sec (5693.11 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum write 1299.23 1299.23 1299.23 0.00 1299.23 1299.23 1299.23 0.00 118.22343 NA NA 0 30 10 1 0 0 1 0 0 1 5368709120 1048576 153600.0 POSIX 0 read 5429.38 5429.38 5429.38 0.00 5429.38 5429.38 5429.38 0.00 28.29054 NA NA 0 30 10 1 0 0 1 0 0 1 5368709120 1048576 153600.0 POSIX 0 Finished : Thu Apr 29 23:26:03 2021
Run mdtest
Create a substantial directory structure.
Use mdtest to create 30K files
mpirun -hostfile /path/to/hostfile_clients -np 10 mdtest -a POSIX -z 0 -F -C -i 1 -n 3334 -e 4096 -d /tmp/daos_dfuse/ -w 4096 -- started at 04/29/2021 23:28:11 -- mdtest-3.4.0+dev was launched with 10 total task(s) on 3 node(s) Command line used: mdtest '-a' 'POSIX' '-z' '0' '-F' '-C' '-i' '1' '-n' '3334' '-e' '4096' '-d' '/tmp/daos_dfuse/' '-w' '4096' Path: /tmp/daos_dfuse FS: 36.5 GiB Used FS: 18.8% Inodes: 2.3 Mi Used Inodes: 5.9% Nodemap: 1001001001 10 tasks, 33340 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 2943.697 2943.674 2943.686 0.006 File stat : 0.000 0.000 0.000 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 1079.858 1079.858 1079.858 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 04/29/2021 23:28:22 --
Run dbench
Install dbench on all client nodes:
sudo yum install dbench
From one of the client node:
dbench --clients-per-process 10 --directory /tmp/daos_dfuse/ --loadfile /usr/share/dbench/client.txt --timelimit 10 10 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 10 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 2 secs failed to create barrier semaphore 9 of 10 processes prepared for launch 0 sec 10 of 10 processes prepared for launch 0 sec releasing clients 0 3 0.00 MB/sec warmup 1 sec latency 826.199 ms 0 7 0.00 MB/sec warmup 2 sec latency 269.284 ms 100 114 288.13 MB/sec execute 1 sec latency 230.428 ms 100 141 184.47 MB/sec execute 2 sec latency 246.159 ms 100 166 147.88 MB/sec execute 3 sec latency 266.298 ms 100 194 133.59 MB/sec execute 4 sec latency 255.767 ms 100 219 121.64 MB/sec execute 5 sec latency 257.980 ms 100 248 117.41 MB/sec execute 6 sec latency 278.191 ms 100 274 112.64 MB/sec execute 7 sec latency 283.694 ms 100 299 107.89 MB/sec execute 8 sec latency 274.483 ms 100 325 104.57 MB/sec execute 9 sec latency 285.639 ms 100 cleanup 10 sec 100 cleanup 11 sec 90 cleanup 12 sec 70 cleanup 13 sec 50 cleanup 14 sec 35 cleanup 15 sec 20 cleanup 16 sec 0 cleanup 17 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 3877 24.215 170.761 Close 3800 0.004 0.022 Qfileinfo 3110 1.488 4.579 WriteX 18750 0.274 6.484 Throughput 104.568 MB/sec 100 clients 10 procs max_latency=285.639 ms
List the dfuse mount point:
# 'testfile' comes from ior run # 'test-dir.0-0' comes from mdtest run # 'clients' comes from dbench run ls /tmp/daos_dfuse clients test-dir.0-0 testfile
DataMover
DataMover using ‘daos’ utility (Single process)
Create Second container
# Create Second container daos container create --pool $DAOS_POOL --type POSIX Successfully created container 158469db-70d2-4a5d-aac9-3c06cbfa7459 export DAOS_CONT2=<cont uuid>
Pool Query before copy:
dmg -o /etc/daos/daos_control.yml pool query --pool $DAOS_POOL Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1 Pool space info: - Target(VOS) count:64 - SCM: Total size: 48 GB Free: 48 GB, min:743 MB, max:744 MB, mean:744 MB - NVMe: Total size: 800 GB Free: 499 GB, min:7.7 GB, max:7.9 GB, mean:7.8 GB Rebuild idle, 0 objs, 0 recs
Move data from POSIX directory into a DAOS container:
Daos 1.2 only supports directory copy if using 'daos filesystem copy'
# moving everything under /tmp/daos_dfuse to new cont $DAOS_CONT2 daos filesystem copy --src /tmp/daos_dfuse/ --dst daos://$DAOS_POOL/$DAOS_CONT2 Successfully copied to DAOS: /
Pool Query to confirm data got copied (Free space has reduced from last pool query):
dmg pool query --pool $DAOS_POOL Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1 Pool space info: - Target(VOS) count:64 - SCM: Total size: 48 GB Free: 47 GB, min:738 MB, max:739 MB, mean:739 MB - NVMe: Total size: 800 GB Free: 338 GB, min:5.1 GB, max:5.5 GB, mean:5.3 GB Rebuild idle, 0 objs, 0 recs
Move data from DAOS container to POSIX directory:
mkdir /tmp/daos_dfuse/daos_container_copy daos filesystem copy --src daos://$DAOS_POOL/$DAOS_CONT2 --dst /tmp/daos_dfuse/daos_container_copy mkdir /tmp/daos_dfuse/daos_cont_copy// failed, File exists Successfully copied to POSIX: /tmp/daos_dfuse/daos_cont_copy/
Pool Query to confirm data got copied:
dmg pool query --pool $DAOS_POOL Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1 Pool space info: - Target(VOS) count:64 - SCM: Total size: 48 GB Free: 47 GB, min:732 MB, max:733 MB, mean:732 MB - NVMe: Total size: 800 GB Free: 128 GB, min:1.8 GB, max:2.3 GB, mean:2.0 GB Rebuild idle, 0 objs, 0 recs
Check data inside the POSIX directories:
ls -latr /tmp/daos_dfuse/ total 157286400 -rw-rw-r-- 1 standan standan 161061273600 Apr 29 23:23 testfile drwxrwxr-x 1 standan standan 64 Apr 29 23:28 test-dir.0-0 drwxrwxr-x 1 standan standan 64 Apr 29 23:30 clients drwxrwxr-x 1 standan standan 64 Apr 30 00:02 daos_container_copy ls -latr /tmp/daos_dfuse/daos_container_copy drwx------ 1 standan standan 64 Apr 30 00:02 daos_dfuse drwx------ 1 standan standan 64 Apr 30 00:11 testfile
DataMover using mpifileutils (allows you to move data in multi-process mode)
Build mpifileutils package
# load mpich module or set it's path in your environment
module load mpi/mpich-x86_64
or
export LD_LIBRARY_PATH=<mpich lib path>:$LD_LIBRARY_PATH
export PATH=<mpich bin path>:$PATH
# install daos-devel, if missing
sudo yum install -y daos-devel
# Build Dependencies
mkdir install
installdir=`pwd`/install
mkdir deps
cd deps
wget https://github.com/hpc/libcircle/releases/download/v0.3/libcircle-0.3.0.tar.gz
wget https://github.com/llnl/lwgrp/releases/download/v1.0.2/lwgrp-1.0.2.tar.gz
wget https://github.com/llnl/dtcmp/releases/download/v1.1.0/dtcmp-1.1.0.tar.gz
tar -zxf libcircle-0.3.0.tar.gz
cd libcircle-0.3.0
./configure --prefix=$installdir
make install
cd ..
tar -zxf lwgrp-1.0.2.tar.gz
cd lwgrp-1.0.2
./configure --prefix=$installdir
make install
cd ..
tar -zxf dtcmp-1.1.0.tar.gz
cd dtcmp-1.1.0
./configure --prefix=$installdir --with-lwgrp=$installdir
make install
cd ..
cd ..
# Build mpifileutils
git clone https://github.com/hpc/mpifileutils
mkdir build
cd build
cmake3 ../mpifileutils/ -DWITH_DTCMP_PREFIX=<path/to/dtcmp/install> -DWITH_LibCircle_PREFIX=<path/to/lib/circle/install> -DWITH_CART_PREFIX=/usr/ -DWITH_DAOS_PREFIX=/usr/ -DCMAKE_INSTALL_PREFIX=<path/where/mpifileutils/need/to/be/installed> -DENABLE_DAOS=ON -DENABLE_LIBARCHIVE=OFF
make install
# On launch node set mpifileutils LD_LIBRARY_PATH and PATH
export LD_LIBRARY_PATH=<mpifileutils/lib/path>:$LD_LIBRARY_PATH
export PATH=<mpifileutils/bin/path>:$PATH
Create Second container
daos container create --pool $DAOS_POOL --type POSIX Successfully created container caf0135c-def8-45a5-bac3-d0b969e67c8b export DAOS_CONT2=<cont uuid>
Move data from POSIX directory into a DAOS container:
mpirun -hostfile /path/to/hostfile -np 16 /path/to/mpifileutils/install/bin/dcp --bufsize 64MB --chunksize 128MB /tmp/daos_dfuse daos://$DAOS_POOL/$DAOS_CONT2/ [2021-04-30T01:16:48] Walking /tmp/daos_dfuse [2021-04-30T01:16:58] Walked 24207 items in 10.030 secs (2413.415 items/sec) ... [2021-04-30T01:17:01] Walked 34245 items in 13.298 secs (2575.138 items/sec) ... [2021-04-30T01:17:01] Walked 34245 items in 13.300 seconds (2574.867 items/sec) [2021-04-30T01:17:01] Copying to / [2021-04-30T01:17:01] Items: 34245 [2021-04-30T01:17:01] Directories: 904 [2021-04-30T01:17:01] Files: 33341 [2021-04-30T01:17:01] Links: 0 [2021-04-30T01:17:01] Data: 150.127 GiB (4.611 MiB per file) [2021-04-30T01:17:01] Creating 904 directories [2021-04-30T01:17:01] Creating 33341 files. [2021-04-30T01:17:02] Copying data. [2021-04-30T01:17:12] Copied 4.049 GiB (3%) in 10.395 secs (398.867 MiB/s) 375 secs left ... [2021-04-30T01:22:37] Copied 8.561 GiB (6%) in 335.113 secs (26.160 MiB/s) 5541 secs left ... [2021-04-30T01:22:37] Copied 150.127 GiB (100%) in 335.113 secs (458.742 MiB/s) done [2021-04-30T01:22:37] Copy data: 150.127 GiB (161197834240 bytes) [2021-04-30T01:22:37] Copy rate: 458.741 MiB/s (161197834240 bytes in 335.114 seconds) [2021-04-30T01:22:37] Syncing data to disk. [2021-04-30T01:22:37] Sync completed in 0.017 seconds. [2021-04-30T01:22:37] Fixing permissions. [2021-04-30T01:22:37] Updated 34245 items in 0.176 seconds (194912.821 items/sec) [2021-04-30T01:22:37] Syncing directory updates to disk. [2021-04-30T01:22:37] Sync completed in 0.012 seconds. [2021-04-30T01:22:37] Started: Apr-30-2021,01:17:01 [2021-04-30T01:22:37] Completed: Apr-30-2021,01:22:37 [2021-04-30T01:22:37] Seconds: 336.013 [2021-04-30T01:22:37] Items: 34245 [2021-04-30T01:22:37] Directories: 904 [2021-04-30T01:22:37] Files: 33341 [2021-04-30T01:22:37] Links: 0 [2021-04-30T01:22:37] Data: 150.127 GiB (161197834240 bytes) [2021-04-30T01:22:37] Rate: 457.513 MiB/s (161197834240 bytes in 336.013 seconds)
Pool Query to verify data got copied (free space should reduce):
dmg pool query --pool $DAOS_POOL Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1 Pool space info: - Target(VOS) count:64 - SCM: Total size: 48 GB Free: 47 GB, min:734 MB, max:735 MB, mean:735 MB - NVMe: Total size: 800 GB Free: 338 GB, min:5.2 GB, max:5.4 GB, mean:5.3 GB Rebuild idle, 0 objs, 0 recs
Move data from DAOS container to POSIX directory:
mkdir /tmp/daos_dfuse/daos_container_copy mpirun -hostfile /path/to/hostfile -np 16 dcp --bufsize 64MB --chunksize 128MB daos://$DAOS_POOL/$DAOS_CONT2/ /tmp/daos_dfuse/daos_container_copy [2021-04-30T01:26:11] Walking / [2021-04-30T01:26:14] Walked 34246 items in 2.648 secs (12930.593 items/sec) ... [2021-04-30T01:26:14] Walked 34246 items in 2.650 seconds (12923.056 items/sec) [2021-04-30T01:26:14] Copying to /tmp/daos_dfuse/daos_container_copy [2021-04-30T01:26:14] Items: 34246 [2021-04-30T01:26:14] Directories: 905 [2021-04-30T01:26:14] Files: 33341 [2021-04-30T01:26:14] Links: 0 [2021-04-30T01:26:14] Data: 150.127 GiB (4.611 MiB per file) [2021-04-30T01:26:14] Creating 905 directories [2021-04-30T01:26:14] Original directory exists, skip the creation: `/tmp/daos_dfuse/daos_container_copy/' (errno=17 File exists) [2021-04-30T01:26:14] Creating 33341 files. [2021-04-30T01:26:19] Copying data. [2021-04-30T01:26:29] Copied 3.819 GiB (3%) in 10.213 secs (382.922 MiB/s) 391 secs left ... [2021-04-30T01:32:02] Copied 150.127 GiB (100%) in 343.861 secs (447.070 MiB/s) done [2021-04-30T01:32:02] Copy data: 150.127 GiB (161197834240 bytes) [2021-04-30T01:32:02] Copy rate: 447.069 MiB/s (161197834240 bytes in 343.862 seconds) [2021-04-30T01:32:02] Syncing data to disk. [2021-04-30T01:32:02] Sync completed in 0.020 seconds. [2021-04-30T01:32:02] Fixing permissions. [2021-04-30T01:32:17] Updated 34162 items in 14.955 secs (2284.295 items/sec) ... [2021-04-30T01:32:17] Updated 34246 items in 14.955 secs (2289.890 items/sec) done [2021-04-30T01:32:17] Updated 34246 items in 14.956 seconds (2289.772 items/sec) [2021-04-30T01:32:17] Syncing directory updates to disk. [2021-04-30T01:32:17] Sync completed in 0.022 seconds. [2021-04-30T01:32:17] Started: Apr-30-2021,01:26:14 [2021-04-30T01:32:17] Completed: Apr-30-2021,01:32:17 [2021-04-30T01:32:17] Seconds: 363.327 [2021-04-30T01:32:17] Items: 34246 [2021-04-30T01:32:17] Directories: 905 [2021-04-30T01:32:17] Files: 33341 [2021-04-30T01:32:17] Links: 0 [2021-04-30T01:32:17] Data: 150.127 GiB (161197834240 bytes) [2021-04-30T01:32:17] Rate: 423.118 MiB/s (161197834240 bytes in 363.327 seconds)
Pool Query to very data was copied:
dmg pool query --pool $DAOS_POOL Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1 Pool space info: - Target(VOS) count:64 - SCM: Total size: 48 GB Free: 47 GB, min:728 MB, max:730 MB, mean:729 MB - NVMe: Total size: 800 GB Free: 176 GB, min:2.6 GB, max:3.0 GB, mean:2.8 GB Rebuild idle, 0 objs, 0 recs
Check data inside the POSIX directories:
ls -latr /tmp/daos_dfuse/ total 157286400 -rw-rw-r-- 1 standan standan 161061273600 Apr 29 23:23 testfile drwxrwxr-x 1 standan standan 64 Apr 29 23:28 test-dir.0-0 drwxrwxr-x 1 standan standan 64 Apr 29 23:30 clients drwxr-xr-x 1 standan standan 64 Apr 30 01:25 daos_container_copy ls -latr /tmp/daos_dfuse/daos_container_copy drwxr-xr-x 1 standan standan 64 Apr 30 01:26 daos_dfuse
For more details on datamover reference: https://github.com/hpc/mpifileutils/blob/master/DAOS-Support.md
Clean Up
Remove one of the copy created using datamover
rm -rf /tmp/daos_dfuse/daos_container_copy
Remove dfuse mountpoint:
# unmount dfuse pdsh -w $CLIENT_NODES 'fusermount3 -uz /tmp/daos_dfuse' # remove mount dir pdsh -w $CLIENT_NODES rm -rf /tmp/daos_dfuse
List containers to be destroyed:
# list containers daos pool list-containers --pool $DAOS_POOL # sample output # sample output cd46cf6e-f886-4682-8077-e3cbcd09b43a caf0135c-def8-45a5-bac3-d0b969e67c8b
Destroy Containers:
# destroy container1 daos container destroy --pool $DAOS_POOL --cont $DAOS_CONT # destroy container2 daos container destroy --pool $DAOS_POOL --cont $DAOS_CONT2
List Pools to be destroyed:
# list pool dmg pool list # sample output Pool UUID Svc Replicas --------- ------------ b22220ea-740d-46bc-84ad-35ed3a28aa31 [1-3]
Destroy Pool:
# destroy pool dmg pool destroy --pool $DAOS_POOL
Stop Agents:
# stop agents pdsh -S -w $CLIENT_NODES "sudo systemctl stop daos_agent"
Stop Servers:
# stop servers pdsh -S -w $SERVER_NODES "sudo systemctl stop daos_server"