Quickstart Centos 7.9 with POSIX
NOTE THESE ARE NOT TO BE APPLIED TO 2.0 TESTING, USE THE QUICKSTARTS IN THE 2.0 ON-LINE DOCUMENTATION
Table of Contents
Introduction
This is a quick start guide to help set up daos and use it with POSIX on Centos 7.9. This document covers any pre-requisites required before starting, pointer to setting up of daos system which consists of installation of daos rpms, setting up of various configuration files needed by daos, getting daos servers and agents up and running. Once the daos system is ready for use, this document can help set up dfuse mount point in order to take advantage of daos support for POSIX, help with some example runs of daos tests and benchmarking tools like ior and mdtest along with some examples of how to move data between a POSIX file system and daos containers (and vise versa) and finally cleaning up your daos setup.
We are using a set of 2 servers each with PMEM and SSDs connected via storage network and 2 client nodes without pmem/ssd but on the storage network. We are also using another node as admin node. This can either be a separate node or any of your client nodes as well. All nodes have a base Centos 7.9 install.
Requirements
Set environment variables for list of servers, clients and admin node.
export ADMIN_NODE=node-1
export SERVER_NODES=node-2,node-3
export CLIENT_NODES=node-4,node-5
export ALL_NODES=$ADMIN_NODE,$SERVER_NODES,$CLIENT_NODESAll nodes have Centos 7.9 installed
Set password less ssh amongst all nodes
Make sure to have sudo privileges on all nodes
Enable IOMMU
If using nvme, enable IOMMU on server nodes: https://daos-stack.github.io/admin/predeployment_check/#enable-iommu-optional
Install pdsh on admin node
# Centos
sudo yum install -y pdshSet-Up
Please refer here for initial set up which consists of rpm installation, generate and set up certificates, setting up config files, starting servers and agents.
Run daos autotest
daos autotest is a smoke test which can be used to test the setup. It performs various activities like connecting a pool, creating and opening a container, reading and writing to the container.
# create pool
dmg pool create --label=pool_1 --size=50G
# sample output
Creating DAOS pool with automatic storage allocation: 50 GB NVMe + 6.00% SCM
Pool created with 6.00% SCM/NVMe ratio
---------------------------------------
UUID : 6af46954-f704-45f0-8b80-115ef855a065
Service Ranks : [1-3]
Storage Ranks : [0-3]
Total Size : 53 GB
SCM : 3.0 GB (750 MB / rank)
NVMe : 50 GB (12 GB / rank)
# assign pool uuid to a variable
export DAOS_POOL=<pool uuid>
# run daos autotest
daos pool autotest --pool $DAOS_POOL
# Sample output
Step Operation Status Time(sec) Comment
0 Initializing DAOS OK 0.000
1 Connecting to pool OK 0.070
2 Creating container OK 0.000 uuid =
3 Opening container OK 0.050
10 Generating 1M S1 layouts OK 4.620
11 Generating 10K SX layouts OK 0.140
20 Inserting 1M 128B values OK 75.130
21 Reading 128B values back OK 71.540
24 Inserting 1M 4KB values OK 109.190
25 Reading 4KB values back OK 103.620
28 Inserting 100K 1MB values OK 413.730
29 Reading 1MB values back OK 461.220
96 Closing container OK 0.040
97 Destroying container OK 0.070
98 Disconnecting from pool OK 0.000
99 Tearing down DAOS OK 0.000
Run cart self_test
On the client run a cart self_test to verify basic network connectivity.
# set env
SHARED_DIR=<shared dir by all nodes>
export FI_UNIVERSE_SIZE=2048
export OFI_INTERFACE=ib0
# generate the attach info file
daos_agent -o /etc/daos/daos_agent.yml -l $SHARED_DIR/daos_agent.log dump-attachinfo -o $SHARED_DIR/daos_server.attach_info_tmp
# selt_test --help for more details on params
# for 4 servers --endpoint 0-3:0-1 ranks:tags.
self_test --path $SHARED_DIR --group-name daos_server --endpoint 0-1:0-1
Adding endpoints:
ranks: 0-1 (# ranks = 2)
tags: 0-1 (# tags = 2)
Warning: No --master-endpoint specified; using this command line application as the master endpoint
Self Test Parameters:
Group name to test against: daos_server
# endpoints: 4
Message sizes: [(200000-BULK_GET 200000-BULK_PUT), (200000-BULK_GET 0-EMPTY), (0-EMPTY 200000-BULK_PUT), (200000-BULK_GET 1000-IOV), (1000-IOV 200000-BULK_PUT), (1000-IOV 1000-IOV), (1000-IOV 0-EMPTY), (0-EMPTY 1000-IOV), (0-EMPTY 0-EMPTY)]
Buffer addresses end with: <Default>
Repetitions per size: 40000
Max inflight RPCs: 1000
CLI [rank=0 pid=40050] Attached daos_server
##################################################
Results for message size (200000-BULK_GET 200000-BULK_PUT) (max_inflight_rpcs = 1000):
Master Endpoint 2:0
-------------------
RPC Bandwidth (MB/sec): 197.56
RPC Throughput (RPCs/sec): 518
RPC Latencies (us):
Min : 38791
25th %: 1695365
Median : 1916632
75th %: 2144087
Max : 2969361
Average: 1907415
Std Dev: 373832.81
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 1889518
0:1 - 1712934
1:0 - 1924995
1:1 - 2110649
##################################################
Results for message size (200000-BULK_GET 0-EMPTY) (max_inflight_rpcs = 1000):
Master Endpoint 2:0
-------------------
RPC Bandwidth (MB/sec): 112.03
RPC Throughput (RPCs/sec): 587
RPC Latencies (us):
Min : 4783
25th %: 1480053
Median : 1688064
75th %: 1897392
Max : 2276555
Average: 1681303
Std Dev: 314999.11
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 2001222
0:1 - 1793990
1:0 - 1385306
1:1 - 1593675
##################################################
Results for message size (0-EMPTY 200000-BULK_PUT) (max_inflight_rpcs = 1000):
Master Endpoint 2:0
-------------------
RPC Bandwidth (MB/sec): 112.12
RPC Throughput (RPCs/sec): 588
RPC Latencies (us):
Min : 6302
25th %: 1063532
Median : 1654468
75th %: 2287784
Max : 3488227
Average: 1680617
Std Dev: 880402.68
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 1251585
0:1 - 1323953
1:0 - 2099173
1:1 - 2043352
##################################################
Results for message size (200000-BULK_GET 1000-IOV) (max_inflight_rpcs = 1000):
Master Endpoint 2:0
-------------------
RPC Bandwidth (MB/sec): 112.54
RPC Throughput (RPCs/sec): 587
RPC Latencies (us):
Min : 5426
25th %: 1395359
Median : 1687404
75th %: 1983402
Max : 2426175
Average: 1681970
Std Dev: 393256.99
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 2077476
0:1 - 1870102
1:0 - 1318136
1:1 - 1529193
##################################################
Results for message size (1000-IOV 200000-BULK_PUT) (max_inflight_rpcs = 1000):
Master Endpoint 2:0
-------------------
RPC Bandwidth (MB/sec): 112.66
RPC Throughput (RPCs/sec): 588
RPC Latencies (us):
Min : 5340
25th %: 442729
Median : 1221371
75th %: 2936906
Max : 3502405
Average: 1681142
Std Dev: 1308472.80
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 3006315
0:1 - 2913808
1:0 - 434763
1:1 - 465469
##################################################
Results for message size (1000-IOV 1000-IOV) (max_inflight_rpcs = 1000):
Master Endpoint 2:0
-------------------
RPC Bandwidth (MB/sec): 80.71
RPC Throughput (RPCs/sec): 42315
RPC Latencies (us):
Min : 1187
25th %: 20187
Median : 23322
75th %: 26833
Max : 30246
Average: 23319
Std Dev: 4339.87
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 26828
0:1 - 26839
1:0 - 20275
1:1 - 20306
##################################################
Results for message size (1000-IOV 0-EMPTY) (max_inflight_rpcs = 1000):
Master Endpoint 2:0
-------------------
RPC Bandwidth (MB/sec): 42.68
RPC Throughput (RPCs/sec): 44758
RPC Latencies (us):
Min : 935
25th %: 15880
Median : 21444
75th %: 28434
Max : 34551
Average: 22035
Std Dev: 7234.26
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 28418
0:1 - 28449
1:0 - 16301
1:1 - 16318
##################################################
Results for message size (0-EMPTY 1000-IOV) (max_inflight_rpcs = 1000):
Master Endpoint 2:0
-------------------
RPC Bandwidth (MB/sec): 42.91
RPC Throughput (RPCs/sec): 44991
RPC Latencies (us):
Min : 789
25th %: 20224
Median : 22195
75th %: 24001
Max : 26270
Average: 21943
Std Dev: 3039.50
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 24017
0:1 - 23987
1:0 - 20279
1:1 - 20309
##################################################
Results for message size (0-EMPTY 0-EMPTY) (max_inflight_rpcs = 1000):
Master Endpoint 2:0
-------------------
RPC Bandwidth (MB/sec): 0.00
RPC Throughput (RPCs/sec): 47807
RPC Latencies (us):
Min : 774
25th %: 16161
Median : 20419
75th %: 25102
Max : 29799
Average: 20633
Std Dev: 5401.96
RPC Failures: 0
Endpoint results (rank:tag - Median Latency (us)):
0:0 - 25103
0:1 - 25099
1:0 - 16401
1:1 - 16421
Run daos_perf
On the client run daos_perf. Requires openmpi3.
(No need to create pool. Daos_perf will take care of it.)
Load mpi or set it in your environment.
# load openmpi module or set it's path in your environment
module load mpi/openmpi3-x86_64
or
export LD_LIBRARY_PATH=<openmpi lib path>:$LD_LIBRARY_PATH
export PATH=<openmpi bin path>:$PATH
Single Process daos_perf run example:
export D_LOG_FILE=/tmp/daos_perf_daos.log
daos_perf -g /etc/daos/daos_control.yml -o 10 -d 10 -a 10 -C 16 -A -R 'U;p F;i=5;p V' -P 1G -N 10G
'U' means UPDATE, 'p' means outputing performance
'F' means FETCH, parameter
"i=5" indicates the FETCH test will repeat 5 times.
'V' means verify correctness of data
‘o’ Number of Objects
‘d’ Number of dkeys
‘a’ Number of akeys
‘A’ Use Array value of akey. Single value is selected by default.
‘P’ Scm Pool size
‘N’ Nvme Pool Size
Test :
DAOS LARGE (full stack, non-replica)
Pool :
79cae139-272e-4ada-922e-98cc968126bf
Parameters :
pool size : SCM: 1024 MB, NVMe: 10240 MB
credits : 16 (sync I/O for -ve)
obj_per_cont : 10 x 1 (procs)
dkey_per_obj : 10
akey_per_dkey : 10
recx_per_akey : 16
value type : array
stride size : 64
zero copy : no
VOS file : <NULL>
Running test=UPDATE
Running UPDATE test (iteration=1)
UPDATE successfully completed:
duration : 0.225937 sec
bandwith : 4.322 MB/sec
rate : 70816.20 IO/sec
latency : 14.121 us (nonsense if credits > 1)
Duration across processes:
MAX duration : 0.225937 sec
MIN duration : 0.225937 sec
Average duration : 0.225937 sec
Completed test=UPDATE
Running test=FETCH
Running FETCH test (iteration=5)
FETCH successfully completed:
duration : 1.190614 sec
bandwith : 4.101 MB/sec
rate : 67192.22 IO/sec
latency : 14.883 us (nonsense if credits > 1)
Duration across processes:
MAX duration : 1.190614 sec
MIN duration : 1.190614 sec
Average duration : 1.190614 sec
Multi-process daos_perf example:
/usr/lib64/openmpi3/bin/orterun --enable-recovery --hostfile /path/to/hostfile_clients --map-by node --mca btl_openib_warn_default_gid_prefix "0" --mca btl "tcp,self" --mca oob "tcp" --mca pml "ob1" --mca btl_tcp_if_include "eth0" --np 4 --tag-output daos_perf -g /etc/daos/daos.yml -o 10 -d 10 -a 10 -C 16 -A -R 'U;p F;i=5;p V' -P 1G -N 10G
[1,0]<stdout>:Test :
[1,0]<stdout>: DAOS LARGE (full stack, non-replica)
[1,0]<stdout>:Pool :
[1,0]<stdout>: 8ce8ae49-57e2-4d6e-b497-b50bbd6375dd
[1,0]<stdout>:Parameters :
[1,0]<stdout>: pool size : SCM: 1024 MB, NVMe: 10240 MB
[1,0]<stdout>: credits : 16 (sync I/O for -ve)
[1,0]<stdout>: obj_per_cont : 10 x 4 (procs)
[1,0]<stdout>: dkey_per_obj : 10
[1,0]<stdout>: akey_per_dkey : 10
[1,0]<stdout>: recx_per_akey : 16
[1,0]<stdout>: value type : array
[1,0]<stdout>: stride size : 64
[1,0]<stdout>: zero copy : no
[1,0]<stdout>: VOS file : <NULL>
[1,0]<stdout>:Running test=UPDATE
[1,0]<stdout>:Running UPDATE test (iteration=1)
[1,2]<stdout>:Running test=UPDATE
[1,2]<stdout>:Running UPDATE test (iteration=1)
[1,1]<stdout>:Running test=UPDATE
[1,1]<stdout>:Running UPDATE test (iteration=1)
[1,3]<stdout>:Running test=UPDATE
[1,3]<stdout>:Running UPDATE test (iteration=1)
[1,2]<stdout>:Completed test=UPDATE
[1,2]<stdout>:Running test=FETCH
[1,2]<stdout>:Running FETCH test (iteration=5)
[1,0]<stdout>:UPDATE successfully completed:
[1,1]<stdout>:Completed test=UPDATE
[1,1]<stdout>:Running test=FETCH
[1,0]<stdout>: duration : 64.994374 sec
[1,0]<stdout>: bandwith : 0.060 MB/sec
[1,0]<stdout>: rate : 984.70 IO/sec
[1,0]<stdout>: latency : 22.106 us (nonsense if credits > 1)
[1,0]<stdout>:Duration across processes:
[1,0]<stdout>: MAX duration : 1.414762 sec
[1,0]<stdout>: MIN duration : 1.064101 sec
[1,3]<stdout>:Completed test=UPDATE
[1,3]<stdout>:Running test=FETCH
[1,3]<stdout>:Running FETCH test (iteration=5)
[1,1]<stdout>:Running FETCH test (iteration=5)
[1,0]<stdout>: Average duration : 1.194978 sec
[1,0]<stdout>:Completed test=UPDATE
[1,0]<stdout>:Running test=FETCH
[1,0]<stdout>:Running FETCH test (iteration=5)
[1,2]<stdout>:Completed test=FETCH
[1,2]<stdout>:Running test=VERIFY
[1,2]<stdout>:Running VERIFY test (iteration=1)
[1,0]<stdout>:FETCH successfully completed:
[1,0]<stdout>: duration : 68.807437 sec
[1,0]<stdout>: bandwith : 0.284 MB/sec
[1,0]<stdout>: rate : 4650.66 IO/sec
[1,0]<stdout>: latency : 16.333 us (nonsense if credits > 1)
[1,0]<stdout>:Duration across processes:
[1,0]<stdout>: MAX duration : 5.226541 sec
[1,0]<stdout>: MIN duration : 3.616002 sec
[1,0]<stdout>: Average duration : 4.349847 sec
[1,0]<stdout>:Completed test=FETCH
[1,0]<stdout>:Running test=VERIFY
[1,0]<stdout>:Running VERIFY test (iteration=1)
[1,3]<stdout>:Completed test=FETCH
[1,3]<stdout>:Running test=VERIFY
[1,1]<stdout>:Completed test=FETCH
[1,1]<stdout>:Running test=VERIFY
[1,1]<stdout>:Running VERIFY test (iteration=1)
[1,3]<stdout>:Running VERIFY test (iteration=1)
[1,0]<stdout>:Completed test=VERIFY
[1,2]<stdout>:Completed test=VERIFY
[1,1]<stdout>:Completed test=VERIFY
[1,3]<stdout>:Completed test=VERIFYRun daos_racer
On the client run daos_racer. Requires openmpi3.
(No need to create pool. Daos_racer will take care of it.)
Steps involve loading of openmpi or setting in environment if not already done. Set log file location and log level, and finally run.
# load openmpi module or set it's path in your environment
module load mpi/openmpi3-x86_64
or
export LD_LIBRARY_PATH=<openmpi lib path>:$LD_LIBRARY_PATH
export PATH=<openmpi bin path>:$PATH
# set environment var
export D_LOG_FILE=/tmp/daos_racer_log
export D_LOG_MASK=ERR
# run daos_racer
/usr/bin/daos_racer -n /etc/daos/daos_control.yml -t 60Run daos_test
On the client run daos_test. Requires openmpi3.
(No need to create pool. Daos_test will take care of it.)
Steps involve loading of openmpi or setting in environment if not already done. Set pool size environment variables, and finally run desired or all daos_tests.
# load openmpi module or set it's path in your environment
module load mpi/openmpi3-x86_64
or
export LD_LIBRARY_PATH=<openmpi lib path>:$LD_LIBRARY_PATH
export PATH=<openmpi bin path>:$PATH
export POOL_SCM_SIZE=8G
export POOL_NVME_SIZE=16G
(Example of Management Test)
/usr/lib64/openmpi3/bin/orterun --mca btl_openib_warn_default_gid_prefix 0 --mca pml ob1 --mca btl tcp,self --mca oob tcp --mca btl_tcp_if_include eth0 -n 1 --hostfile /path/to/hostfile_clients -x D_LOG_FILE=/tmp/daos_test.log --map-by node -x D_LOG_MASK=DEBUG -x DD_MASK=mgmt,io,md,epc,rebuild -x COVFILE=/tmp/test.cov /usr/bin/daos_test -n /etc/daos/daos_control.yml -m
=================
DAOS management tests..
=====================
[==========] Running 5 test(s).
[ RUN ] MGMT1: create/destroy pool on all tgts
creating pool synchronously ... success uuid = 6e3dbfee-ee2e-47ec-8b3c-6a42b62d7573
destroying pool synchronously ... success
[ OK ] MGMT1: create/destroy pool on all tgts
[ RUN ] MGMT2: create/destroy pool on all tgts (async)
async not supported[ OK ] MGMT2: create/destroy pool on all tgts (async)
[ RUN ] MGMT3: list-pools with no pools in sys
success t0: output npools=0
verifying pools[0..10], nfilled=0
success t1: pools[] over-sized
success t2: npools=0, non-NULL pools[] rc=0
success t3: in &npools NULL, -DER_INVAL
success
[ OK ] MGMT3: list-pools with no pools in sys
[ RUN ] MGMT4: list-pools with multiple pools in sys
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool 24aea230-2366-45a6-90e8-63ab36e58f8b
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool cb936b0c-c32f-4465-86f9-d18aa94a5338
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool 02f0d551-843d-4677-9905-696205c1fb5c
success t0: output npools=4
verifying pools[0..14], nfilled=4
pool 24aea230-2366-45a6-90e8-63ab36e58f8b found in list result
pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e found in list result
pool cb936b0c-c32f-4465-86f9-d18aa94a5338 found in list result
pool 02f0d551-843d-4677-9905-696205c1fb5c found in list result
success t1: pools[] over-sized
success t2: npools=0, non-NULL pools[] rc=-1016
success t3: in &npools NULL, -DER_INVAL
verifying pools[0..4], nfilled=4
pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e found in list result
pool cb936b0c-c32f-4465-86f9-d18aa94a5338 found in list result
pool 02f0d551-843d-4677-9905-696205c1fb5c found in list result
pool 24aea230-2366-45a6-90e8-63ab36e58f8b found in list result
success t4: pools[] exact length
verifying pools[0..3], nfilled=0
success t5: pools[] under-sized
success
teardown: destroyed pool 24aea230-2366-45a6-90e8-63ab36e58f8b
teardown: destroyed pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e
teardown: destroyed pool cb936b0c-c32f-4465-86f9-d18aa94a5338
teardown: destroyed pool 02f0d551-843d-4677-9905-696205c1fb5c
[ OK ] MGMT4: list-pools with multiple pools in sys
[ RUN ] MGMT5: retry MGMT_POOL_{CREATE,DESETROY} upon errors
setting DAOS_POOL_CREATE_FAIL_CORPC ... success
creating pool synchronously ... success uuid = a93bd58f-1c9a-4adf-8a96-e366e6f25c6a
destroying pool synchronously ... success
[ OK ] MGMT5: retry MGMT_POOL_{CREATE,DESETROY} upon errors
[==========] 5 test(s) run.
[ PASSED ] 5 test(s).
============ Summary src/tests/suite/daos_test.c
OK - NO TEST FAILURESAll DAOS TESTS
=============================
Tests: Use one of these arg(s) for specific test
daos_test -m|--mgmt
daos_test -p|--daos_pool_tests
daos_test -c|--daos_container_tests
daos_test -C|--capa
daos_test -U|--dedup
daos_test -z|--checksum
daos_test -Z|--ec_aggregation
daos_test -t|--base_tx
daos_test -T|--dist_tx
daos_test -i|--daos_io_tests
daos_test -x|--epoch_io
daos_test -A|--array
daos_test -D|--daos_array
daos_test -K|--daos_kv
daos_test -d|--degraded
daos_test -e|--daos_epoch_tests
daos_test -o|--daos_epoch_recovery_tests
daos_test -V|--verify_consistency
daos_test -R|--MD_replication_tests
daos_test -O|--oid_alloc
daos_test -r|--rebuild
daos_test -v|--rebuild_simple
daos_test -S|--rebuild_ec
daos_test -X|--degrade_ec
daos_test -b|--drain_simple
daos_test -B|--extend_simple
daos_test -N|--nvme_recovery
daos_test -a|--daos_all_tests
Default <daos_tests> runs all tests
Create pool, posix container and then mount the container via fuse
Steps:
Create a pool.
Create a posix type container.
Make directory for dfuse mount point.
Mount dfuse.
Verify dfuse got mounted.
# create pool
dmg pool create --label=pool_2 --size=800G
Creating DAOS pool with automatic storage allocation: 800 GB NVMe + 6.00% SCM
Pool created with 6.00% SCM/NVMe ratio
---------------------------------------
UUID : b22220ea-740d-46bc-84ad-35ed3a28aa31
Service Ranks : [1-3]
Storage Ranks : [0-7]
Total Size : 848 GB
SCM : 48 GB (6.0 GB / rank)
NVMe : 800 GB (100 GB / rank)
# List pool
dmg pool list
Pool UUID Svc Replicas
--------- ------------
b22220ea-740d-46bc-84ad-35ed3a28aa31 [1-3]
# assign pool uuid to DAOS_POOL variable
export DAOS_POOL=b22220ea-740d-46bc-84ad-35ed3a28aa31
# Pool Query
dmg pool query $DAOS_POOL
Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
Total size: 48 GB
Free: 48 GB, min:750 MB, max:750 MB, mean:750 MB
- NVMe:
Total size: 800 GB
Free: 660 GB, min:10 GB, max:10 GB, mean:10 GB
Rebuild idle, 0 objs, 0 recs
# create container of type POSIX
daos container create --pool=$DAOS_POOL --type=POSIX
Successfully created container cd46cf6e-f886-4682-8077-e3cbcd09b43a
# assign container uuid to DAOS_CONT variable
export DAOS_CONT=cd46cf6e-f886-4682-8077-e3cbcd09b43a
# create dir for dfuse mountpoint on all client nodes
pdsh -w $CLIENT_NODES 'mkdir /tmp/daos_dfuse'
# mount dfuse
pdsh -w $CLIENT_NODES dfuse --pool=$DAOS_POOL --cont=$DAOS_CONT --mountpoint=/tmp/daos_dfuse/
#verify dfuse got mounted.
pdsh -w $CLIENT_NODES df /tmp/daos_dfuse
wolf-86: Filesystem 1K-blocks Used Available Use% Mounted on
wolf-86: dfuse 828125000 136841868 691283133 17% /tmp/daos_dfuse
wolf-88: Filesystem 1K-blocks Used Available Use% Mounted on
wolf-88: dfuse 828125000 136841868 691283133 17% /tmp/daos_dfuse
wolf-87: Filesystem 1K-blocks Used Available Use% Mounted on
wolf-87: dfuse 828125000 136841868 691283133 17% /tmp/daos_dfuseRun IOR
Use IOR to generate an HPC type POSIX I/O load to the POSIX container.
Build ior
git clone https://github.com/hpc/ior.git
cd ior
./bootstrap
mkdir build;cd build
../configure --with-daos=/usr --prefix=<your dir>
make
make installAdd <your dir>/lib to LD_LIBRARY_PATh and <your dir>/bin to PATH
Or
On all client nodes:
sudo yum install ior
Use ior to write and read around 150G of data
# load mpich module or set it's path in your environment
module load mpi/mpich-x86_64
or
export LD_LIBRARY_PATH=<mpich lib path>:$LD_LIBRARY_PATH
export PATH=<mpich bin path>:$PATH
mpirun -hostfile /path/to/hostfile_clients -np 30 ior -a POSIX -b 5G -t 1M -v -W -w -r -R -i 1 -k -o /tmp/daos_dfuse/testfile
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began : Thu Apr 29 23:23:09 2021
Command line : ior -a POSIX -b 5G -t 1M -v -W -w -r -R -i 1 -k -o /tmp/daos_dfuse/testfile
Machine : Linux wolf-86.wolf.hpdd.intel.com
Start time skew across all tasks: 0.00 sec
TestID : 0
StartTime : Thu Apr 29 23:23:09 2021
Path : /tmp/daos_dfuse/testfile
FS : 789.8 GiB Used FS: 16.5% Inodes: -0.0 Mi Used Inodes: 0.0%
Participating tasks : 30
Options:
api : POSIX
apiVersion :
test filename : /tmp/daos_dfuse/testfile
access : single-shared-file
type : independent
segments : 1
ordering in a file : sequential
ordering inter file : no tasks offsets
nodes : 3
tasks : 30
clients per node : 10
repetitions : 1
xfersize : 1 MiB
blocksize : 5 GiB
aggregate filesize : 150 GiB
verbose : 1
Results:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
Commencing write performance test: Thu Apr 29 23:23:09 2021
write 1299.23 1299.84 0.022917 5242880 1024.00 10.79 118.17 0.000377 118.22 0
Verifying contents of the file(s) just written.
Thu Apr 29 23:25:07 2021
Commencing read performance test: Thu Apr 29 23:25:35 2021
read 5429 5431 0.005523 5242880 1024.00 0.012188 28.28 0.000251 28.29 0
Max Write: 1299.23 MiB/sec (1362.35 MB/sec)
Max Read: 5429.38 MiB/sec (5693.11 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum
write 1299.23 1299.23 1299.23 0.00 1299.23 1299.23 1299.23 0.00 118.22343 NA NA 0 30 10 1 0 0 1 0 0 1 5368709120 1048576 153600.0 POSIX 0
read 5429.38 5429.38 5429.38 0.00 5429.38 5429.38 5429.38 0.00 28.29054 NA NA 0 30 10 1 0 0 1 0 0 1 5368709120 1048576 153600.0 POSIX 0
Finished : Thu Apr 29 23:26:03 2021Run mdtest
Create a substantial directory structure.
Use mdtest to create 30K files
mpirun -hostfile /path/to/hostfile_clients -np 10 mdtest -a POSIX -z 0 -F -C -i 1 -n 3334 -e 4096 -d /tmp/daos_dfuse/ -w 4096
-- started at 04/29/2021 23:28:11 --
mdtest-3.4.0+dev was launched with 10 total task(s) on 3 node(s)
Command line used: mdtest '-a' 'POSIX' '-z' '0' '-F' '-C' '-i' '1' '-n' '3334' '-e' '4096' '-d' '/tmp/daos_dfuse/' '-w' '4096'
Path: /tmp/daos_dfuse
FS: 36.5 GiB Used FS: 18.8% Inodes: 2.3 Mi Used Inodes: 5.9%
Nodemap: 1001001001
10 tasks, 33340 files
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 2943.697 2943.674 2943.686 0.006
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
Tree creation : 1079.858 1079.858 1079.858 0.000
Tree removal : 0.000 0.000 0.000 0.000
-- finished at 04/29/2021 23:28:22 --Run dbench
Install dbench on all client nodes:
sudo yum install dbench
From one of the client node:
dbench --clients-per-process 10 --directory /tmp/daos_dfuse/ --loadfile /usr/share/dbench/client.txt --timelimit 10 10
dbench version 4.00 - Copyright Andrew Tridgell 1999-2004
Running for 10 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 2 secs
failed to create barrier semaphore
9 of 10 processes prepared for launch 0 sec
10 of 10 processes prepared for launch 0 sec
releasing clients
0 3 0.00 MB/sec warmup 1 sec latency 826.199 ms
0 7 0.00 MB/sec warmup 2 sec latency 269.284 ms
100 114 288.13 MB/sec execute 1 sec latency 230.428 ms
100 141 184.47 MB/sec execute 2 sec latency 246.159 ms
100 166 147.88 MB/sec execute 3 sec latency 266.298 ms
100 194 133.59 MB/sec execute 4 sec latency 255.767 ms
100 219 121.64 MB/sec execute 5 sec latency 257.980 ms
100 248 117.41 MB/sec execute 6 sec latency 278.191 ms
100 274 112.64 MB/sec execute 7 sec latency 283.694 ms
100 299 107.89 MB/sec execute 8 sec latency 274.483 ms
100 325 104.57 MB/sec execute 9 sec latency 285.639 ms
100 cleanup 10 sec
100 cleanup 11 sec
90 cleanup 12 sec
70 cleanup 13 sec
50 cleanup 14 sec
35 cleanup 15 sec
20 cleanup 16 sec
0 cleanup 17 sec
Operation Count AvgLat MaxLat
----------------------------------------
NTCreateX 3877 24.215 170.761
Close 3800 0.004 0.022
Qfileinfo 3110 1.488 4.579
WriteX 18750 0.274 6.484
Throughput 104.568 MB/sec 100 clients 10 procs max_latency=285.639 ms
List the dfuse mount point:
# 'testfile' comes from ior run
# 'test-dir.0-0' comes from mdtest run
# 'clients' comes from dbench run
ls /tmp/daos_dfuse
clients test-dir.0-0 testfileDataMover
DataMover using ‘daos’ utility (Single process)
Create Second container
# Create Second container
daos container create --pool $DAOS_POOL --type POSIX
Successfully created container 158469db-70d2-4a5d-aac9-3c06cbfa7459
export DAOS_CONT2=<cont uuid>
Pool Query before copy:
dmg -o /etc/daos/daos_control.yml pool query --pool $DAOS_POOL
Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
Total size: 48 GB
Free: 48 GB, min:743 MB, max:744 MB, mean:744 MB
- NVMe:
Total size: 800 GB
Free: 499 GB, min:7.7 GB, max:7.9 GB, mean:7.8 GB
Rebuild idle, 0 objs, 0 recs
Move data from POSIX directory into a DAOS container:
Daos 1.2 only supports directory copy if using 'daos filesystem copy'
# moving everything under /tmp/daos_dfuse to new cont $DAOS_CONT2
daos filesystem copy --src /tmp/daos_dfuse/ --dst daos://$DAOS_POOL/$DAOS_CONT2
Successfully copied to DAOS: /
Pool Query to confirm data got copied (Free space has reduced from last pool query):
dmg pool query $DAOS_POOL
Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
Total size: 48 GB
Free: 47 GB, min:738 MB, max:739 MB, mean:739 MB
- NVMe:
Total size: 800 GB
Free: 338 GB, min:5.1 GB, max:5.5 GB, mean:5.3 GB
Rebuild idle, 0 objs, 0 recs
Move data from DAOS container to POSIX directory:
mkdir /tmp/daos_dfuse/daos_container_copy
daos filesystem copy --src daos://$DAOS_POOL/$DAOS_CONT2 --dst /tmp/daos_dfuse/daos_container_copy
mkdir /tmp/daos_dfuse/daos_container_copy
mkdir: cannot create directory ‘/tmp/daos_dfuse/daos_container_copy’: File exists
Pool Query to confirm data got copied:
dmg pool query $DAOS_POOL
Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
Total size: 48 GB
Free: 47 GB, min:732 MB, max:733 MB, mean:732 MB
- NVMe:
Total size: 800 GB
Free: 128 GB, min:1.8 GB, max:2.3 GB, mean:2.0 GB
Rebuild idle, 0 objs, 0 recs
Check data inside the POSIX directories:
ls -latr /tmp/daos_dfuse/
total 157286400
-rw-rw-r-- 1 standan standan 161061273600 Apr 29 23:23 testfile
drwxrwxr-x 1 standan standan 64 Apr 29 23:28 test-dir.0-0
drwxrwxr-x 1 standan standan 64 Apr 29 23:30 clients
drwxrwxr-x 1 standan standan 64 Apr 30 00:02 daos_container_copy
ls -latr /tmp/daos_dfuse/daos_container_copy
drwx------ 1 standan standan 64 Apr 30 00:02 daos_dfuse
drwx------ 1 standan standan 64 Apr 30 00:11 testfileDataMover using mpifileutils (allows you to move data in multi-process mode)
Build mpifileutils package
# load mpich module or set it's path in your environment
module load mpi/mpich-x86_64
or
export LD_LIBRARY_PATH=<mpich lib path>:$LD_LIBRARY_PATH
export PATH=<mpich bin path>:$PATH
# install daos-devel, if missing
sudo yum install -y daos-devel
# Build Dependencies
mkdir install
installdir=`pwd`/install
mkdir deps
cd deps
wget https://github.com/hpc/libcircle/releases/download/v0.3/libcircle-0.3.0.tar.gz
wget https://github.com/llnl/lwgrp/releases/download/v1.0.2/lwgrp-1.0.2.tar.gz
wget https://github.com/llnl/dtcmp/releases/download/v1.1.0/dtcmp-1.1.0.tar.gz
tar -zxf libcircle-0.3.0.tar.gz
cd libcircle-0.3.0
./configure --prefix=$installdir
make install
cd ..
tar -zxf lwgrp-1.0.2.tar.gz
cd lwgrp-1.0.2
./configure --prefix=$installdir
make install
cd ..
tar -zxf dtcmp-1.1.0.tar.gz
cd dtcmp-1.1.0
./configure --prefix=$installdir --with-lwgrp=$installdir
make install
cd ..
cd ..
# Build mpifileutils
git clone https://github.com/hpc/mpifileutils
mkdir build
cd build
cmake3 ../mpifileutils/ -DWITH_DTCMP_PREFIX=<path/to/dtcmp/install> -DWITH_LibCircle_PREFIX=<path/to/lib/circle/install> -DWITH_CART_PREFIX=/usr/ -DWITH_DAOS_PREFIX=/usr/ -DCMAKE_INSTALL_PREFIX=<path/where/mpifileutils/need/to/be/installed> -DENABLE_DAOS=ON -DENABLE_LIBARCHIVE=OFF
make install
# On launch node set mpifileutils LD_LIBRARY_PATH and PATH
export LD_LIBRARY_PATH=<mpifileutils/lib/path>:$LD_LIBRARY_PATH
export PATH=<mpifileutils/bin/path>:$PATH
Create Second container
daos container create --pool $DAOS_POOL --type POSIX
Successfully created container caf0135c-def8-45a5-bac3-d0b969e67c8b
export DAOS_CONT2=<cont uuid>
Move data from POSIX directory into a DAOS container:
mpirun -hostfile /path/to/hostfile -np 16 /path/to/mpifileutils/install/bin/dcp --bufsize 64MB --chunksize 128MB /tmp/daos_dfuse daos://$DAOS_POOL/$DAOS_CONT2/
[2021-04-30T01:16:48] Walking /tmp/daos_dfuse
[2021-04-30T01:16:58] Walked 24207 items in 10.030 secs (2413.415 items/sec) ...
[2021-04-30T01:17:01] Walked 34245 items in 13.298 secs (2575.138 items/sec) ...
[2021-04-30T01:17:01] Walked 34245 items in 13.300 seconds (2574.867 items/sec)
[2021-04-30T01:17:01] Copying to /
[2021-04-30T01:17:01] Items: 34245
[2021-04-30T01:17:01] Directories: 904
[2021-04-30T01:17:01] Files: 33341
[2021-04-30T01:17:01] Links: 0
[2021-04-30T01:17:01] Data: 150.127 GiB (4.611 MiB per file)
[2021-04-30T01:17:01] Creating 904 directories
[2021-04-30T01:17:01] Creating 33341 files.
[2021-04-30T01:17:02] Copying data.
[2021-04-30T01:17:12] Copied 4.049 GiB (3%) in 10.395 secs (398.867 MiB/s) 375 secs left ...
[2021-04-30T01:22:37] Copied 8.561 GiB (6%) in 335.113 secs (26.160 MiB/s) 5541 secs left ...
[2021-04-30T01:22:37] Copied 150.127 GiB (100%) in 335.113 secs (458.742 MiB/s) done
[2021-04-30T01:22:37] Copy data: 150.127 GiB (161197834240 bytes)
[2021-04-30T01:22:37] Copy rate: 458.741 MiB/s (161197834240 bytes in 335.114 seconds)
[2021-04-30T01:22:37] Syncing data to disk.
[2021-04-30T01:22:37] Sync completed in 0.017 seconds.
[2021-04-30T01:22:37] Fixing permissions.
[2021-04-30T01:22:37] Updated 34245 items in 0.176 seconds (194912.821 items/sec)
[2021-04-30T01:22:37] Syncing directory updates to disk.
[2021-04-30T01:22:37] Sync completed in 0.012 seconds.
[2021-04-30T01:22:37] Started: Apr-30-2021,01:17:01
[2021-04-30T01:22:37] Completed: Apr-30-2021,01:22:37
[2021-04-30T01:22:37] Seconds: 336.013
[2021-04-30T01:22:37] Items: 34245
[2021-04-30T01:22:37] Directories: 904
[2021-04-30T01:22:37] Files: 33341
[2021-04-30T01:22:37] Links: 0
[2021-04-30T01:22:37] Data: 150.127 GiB (161197834240 bytes)
[2021-04-30T01:22:37] Rate: 457.513 MiB/s (161197834240 bytes in 336.013 seconds)
Pool Query to verify data got copied (free space should reduce):
dmg pool query --pool $DAOS_POOL
Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
Total size: 48 GB
Free: 47 GB, min:734 MB, max:735 MB, mean:735 MB
- NVMe:
Total size: 800 GB
Free: 338 GB, min:5.2 GB, max:5.4 GB, mean:5.3 GB
Rebuild idle, 0 objs, 0 recs
Move data from DAOS container to POSIX directory:
mkdir /tmp/daos_dfuse/daos_container_copy
mpirun -hostfile /path/to/hostfile -np 16 dcp --bufsize 64MB --chunksize 128MB daos://$DAOS_POOL/$DAOS_CONT2/ /tmp/daos_dfuse/daos_container_copy
[2021-04-30T01:26:11] Walking /
[2021-04-30T01:26:14] Walked 34246 items in 2.648 secs (12930.593 items/sec) ...
[2021-04-30T01:26:14] Walked 34246 items in 2.650 seconds (12923.056 items/sec)
[2021-04-30T01:26:14] Copying to /tmp/daos_dfuse/daos_container_copy
[2021-04-30T01:26:14] Items: 34246
[2021-04-30T01:26:14] Directories: 905
[2021-04-30T01:26:14] Files: 33341
[2021-04-30T01:26:14] Links: 0
[2021-04-30T01:26:14] Data: 150.127 GiB (4.611 MiB per file)
[2021-04-30T01:26:14] Creating 905 directories
[2021-04-30T01:26:14] Original directory exists, skip the creation: `/tmp/daos_dfuse/daos_container_copy/' (errno=17 File exists)
[2021-04-30T01:26:14] Creating 33341 files.
[2021-04-30T01:26:19] Copying data.
[2021-04-30T01:26:29] Copied 3.819 GiB (3%) in 10.213 secs (382.922 MiB/s) 391 secs left ...
[2021-04-30T01:32:02] Copied 150.127 GiB (100%) in 343.861 secs (447.070 MiB/s) done
[2021-04-30T01:32:02] Copy data: 150.127 GiB (161197834240 bytes)
[2021-04-30T01:32:02] Copy rate: 447.069 MiB/s (161197834240 bytes in 343.862 seconds)
[2021-04-30T01:32:02] Syncing data to disk.
[2021-04-30T01:32:02] Sync completed in 0.020 seconds.
[2021-04-30T01:32:02] Fixing permissions.
[2021-04-30T01:32:17] Updated 34162 items in 14.955 secs (2284.295 items/sec) ...
[2021-04-30T01:32:17] Updated 34246 items in 14.955 secs (2289.890 items/sec) done
[2021-04-30T01:32:17] Updated 34246 items in 14.956 seconds (2289.772 items/sec)
[2021-04-30T01:32:17] Syncing directory updates to disk.
[2021-04-30T01:32:17] Sync completed in 0.022 seconds.
[2021-04-30T01:32:17] Started: Apr-30-2021,01:26:14
[2021-04-30T01:32:17] Completed: Apr-30-2021,01:32:17
[2021-04-30T01:32:17] Seconds: 363.327
[2021-04-30T01:32:17] Items: 34246
[2021-04-30T01:32:17] Directories: 905
[2021-04-30T01:32:17] Files: 33341
[2021-04-30T01:32:17] Links: 0
[2021-04-30T01:32:17] Data: 150.127 GiB (161197834240 bytes)
[2021-04-30T01:32:17] Rate: 423.118 MiB/s (161197834240 bytes in 363.327 seconds)
Pool Query to very data was copied:
dmg pool query --pool $DAOS_POOL
Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
Total size: 48 GB
Free: 47 GB, min:728 MB, max:730 MB, mean:729 MB
- NVMe:
Total size: 800 GB
Free: 176 GB, min:2.6 GB, max:3.0 GB, mean:2.8 GB
Rebuild idle, 0 objs, 0 recs
Check data inside the POSIX directories:
ls -latr /tmp/daos_dfuse/
total 157286400
-rw-rw-r-- 1 standan standan 161061273600 Apr 29 23:23 testfile
drwxrwxr-x 1 standan standan 64 Apr 29 23:28 test-dir.0-0
drwxrwxr-x 1 standan standan 64 Apr 29 23:30 clients
drwxr-xr-x 1 standan standan 64 Apr 30 01:25 daos_container_copy
ls -latr /tmp/daos_dfuse/daos_container_copy
drwxr-xr-x 1 standan standan 64 Apr 30 01:26 daos_dfuse For more details on datamover reference: https://github.com/hpc/mpifileutils/blob/master/DAOS-Support.md
Clean Up
Remove one of the copy created using datamover
rm -rf /tmp/daos_dfuse/daos_container_copyRemove dfuse mountpoint:
# unmount dfuse
pdsh -w $CLIENT_NODES 'fusermount3 -uz /tmp/daos_dfuse'
# remove mount dir
pdsh -w $CLIENT_NODES rm -rf /tmp/daos_dfuseList containers to be destroyed:
# list containers
daos pool list-containers --pool $DAOS_POOL # sample output
# sample output
cd46cf6e-f886-4682-8077-e3cbcd09b43a
caf0135c-def8-45a5-bac3-d0b969e67c8bDestroy Containers:
# destroy container1
daos container destroy --pool $DAOS_POOL --cont $DAOS_CONT
# destroy container2
daos container destroy --pool $DAOS_POOL --cont $DAOS_CONT2List Pools to be destroyed:
# list pool
dmg pool list
# sample output
Pool UUID Svc Replicas
--------- ------------
b22220ea-740d-46bc-84ad-35ed3a28aa31 [1-3] Destroy Pool:
# destroy pool
dmg pool destroy --pool $DAOS_POOLStop Agents:
# stop agents
pdsh -S -w $CLIENT_NODES "sudo systemctl stop daos_agent"Stop Servers:
# stop servers
pdsh -S -w $SERVER_NODES "sudo systemctl stop daos_server"
In the provided server config, the settings under "engines" should be indented. Otherwise, the config isn't parsed correctly. (for me anyway)