Quickstart Centos 7.9 with POSIX

Owned by Saurabh Tandan
Last updated: Dec 20, 2021 by Craig Durfey (Deactivated)
25 min read

NOTE THESE ARE NOT TO BE APPLIED TO 2.0 TESTING, USE THE QUICKSTARTS IN THE 2.0 ON-LINE DOCUMENTATION

Table of Contents

Introduction

This is a quick start guide to help set up daos and use it with POSIX on Centos 7.9. This document covers any pre-requisites required before starting, pointer to setting up of daos system which consists of installation of daos rpms, setting up of various configuration files needed by daos, getting daos servers and agents up and running. Once the daos system is ready for use, this document can help set up dfuse mount point in order to take advantage of daos support for POSIX, help with some example runs of daos tests and benchmarking tools like ior and mdtest along with some examples of how to move data between a POSIX file system and daos containers (and vise versa) and finally cleaning up your daos setup.

We are using a set of 2 servers each with PMEM and SSDs connected via storage network and 2 client nodes without pmem/ssd but on the storage network. We are also using another node as admin node. This can either be a separate node or any of your client nodes as well. All nodes have a base Centos 7.9 install.

Requirements

Set environment variables for list of servers, clients and admin node.

export ADMIN_NODE=node-1
export SERVER_NODES=node-2,node-3
export CLIENT_NODES=node-4,node-5
export ALL_NODES=$ADMIN_NODE,$SERVER_NODES,$CLIENT_NODES

All nodes have Centos 7.9 installed

Set password less ssh amongst all nodes

Make sure to have sudo privileges on all nodes

Enable IOMMU

If using nvme, enable IOMMU on server nodes: https://daos-stack.github.io/admin/predeployment_check/#enable-iommu-optional

Install pdsh on admin node

# Centos
sudo yum install -y pdsh

Set-Up

Please refer here for initial set up which consists of rpm installation, generate and set up certificates, setting up config files, starting servers and agents.

Run daos autotest

daos autotest is a smoke test which can be used to test the setup. It performs various activities like connecting a pool, creating and opening a container, reading and writing to the container.

# create pool
dmg pool create --label=pool_1 --size=50G

# sample output
Creating DAOS pool with automatic storage allocation: 50 GB NVMe + 6.00% SCM
Pool created with 6.00% SCM/NVMe ratio
---------------------------------------
  UUID          : 6af46954-f704-45f0-8b80-115ef855a065
  Service Ranks : [1-3]                               
  Storage Ranks : [0-3]                               
  Total Size    : 53 GB                               
  SCM           : 3.0 GB (750 MB / rank)              
  NVMe          : 50 GB (12 GB / rank)                

# assign pool uuid to a variable
export DAOS_POOL=<pool uuid>

# run daos autotest
daos pool autotest --pool $DAOS_POOL

# Sample output
Step Operation               Status Time(sec) Comment
  0  Initializing DAOS          OK      0.000  
  1  Connecting to pool         OK      0.070  
  2  Creating container         OK      0.000  uuid = 
  3  Opening container          OK      0.050  
 10  Generating 1M S1 layouts   OK      4.620  
 11  Generating 10K SX layouts  OK      0.140  
 20  Inserting 1M 128B values   OK     75.130  
 21  Reading 128B values back   OK     71.540  
 24  Inserting 1M 4KB values    OK    109.190  
 25  Reading 4KB values back    OK    103.620  
 28  Inserting 100K 1MB values  OK    413.730  
 29  Reading 1MB values back    OK    461.220  
 96  Closing container          OK      0.040  
 97  Destroying container       OK      0.070  
 98  Disconnecting from pool    OK      0.000  
 99  Tearing down DAOS          OK      0.000



Run cart self_test

On the client run a cart self_test to verify basic network connectivity.

# set env
SHARED_DIR=<shared dir by all nodes>
export FI_UNIVERSE_SIZE=2048
export OFI_INTERFACE=ib0

# generate the attach info file
daos_agent -o /etc/daos/daos_agent.yml -l $SHARED_DIR/daos_agent.log dump-attachinfo -o $SHARED_DIR/daos_server.attach_info_tmp

# selt_test --help for more details on params
# for 4 servers --endpoint 0-3:0-1 ranks:tags.
self_test --path $SHARED_DIR --group-name daos_server --endpoint 0-1:0-1

Adding endpoints:
  ranks: 0-1 (# ranks = 2)
  tags: 0-1 (# tags = 2)
Warning: No --master-endpoint specified; using this command line application as the master endpoint
Self Test Parameters:
  Group name to test against: daos_server
  # endpoints:                4
  Message sizes:              [(200000-BULK_GET 200000-BULK_PUT), (200000-BULK_GET 0-EMPTY), (0-EMPTY 200000-BULK_PUT), (200000-BULK_GET 1000-IOV), (1000-IOV 200000-BULK_PUT), (1000-IOV 1000-IOV), (1000-IOV 0-EMPTY), (0-EMPTY 1000-IOV), (0-EMPTY 0-EMPTY)]
  Buffer addresses end with:  <Default>
  Repetitions per size:       40000
  Max inflight RPCs:          1000

CLI [rank=0 pid=40050]	Attached daos_server
##################################################
Results for message size (200000-BULK_GET 200000-BULK_PUT) (max_inflight_rpcs = 1000):

Master Endpoint 2:0
-------------------
	RPC Bandwidth (MB/sec): 197.56
	RPC Throughput (RPCs/sec): 518
	RPC Latencies (us):
		Min    : 38791
		25th  %: 1695365
		Median : 1916632
		75th  %: 2144087
		Max    : 2969361
		Average: 1907415
		Std Dev: 373832.81
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 1889518
		0:1 - 1712934
		1:0 - 1924995
		1:1 - 2110649

##################################################
Results for message size (200000-BULK_GET 0-EMPTY) (max_inflight_rpcs = 1000):

Master Endpoint 2:0
-------------------
	RPC Bandwidth (MB/sec): 112.03
	RPC Throughput (RPCs/sec): 587
	RPC Latencies (us):
		Min    : 4783
		25th  %: 1480053
		Median : 1688064
		75th  %: 1897392
		Max    : 2276555
		Average: 1681303
		Std Dev: 314999.11
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 2001222
		0:1 - 1793990
		1:0 - 1385306
		1:1 - 1593675

##################################################
Results for message size (0-EMPTY 200000-BULK_PUT) (max_inflight_rpcs = 1000):

Master Endpoint 2:0
-------------------
	RPC Bandwidth (MB/sec): 112.12
	RPC Throughput (RPCs/sec): 588
	RPC Latencies (us):
		Min    : 6302
		25th  %: 1063532
		Median : 1654468
		75th  %: 2287784
		Max    : 3488227
		Average: 1680617
		Std Dev: 880402.68
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 1251585
		0:1 - 1323953
		1:0 - 2099173
		1:1 - 2043352

##################################################
Results for message size (200000-BULK_GET 1000-IOV) (max_inflight_rpcs = 1000):

Master Endpoint 2:0
-------------------
	RPC Bandwidth (MB/sec): 112.54
	RPC Throughput (RPCs/sec): 587
	RPC Latencies (us):
		Min    : 5426
		25th  %: 1395359
		Median : 1687404
		75th  %: 1983402
		Max    : 2426175
		Average: 1681970
		Std Dev: 393256.99
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 2077476
		0:1 - 1870102
		1:0 - 1318136
		1:1 - 1529193

##################################################
Results for message size (1000-IOV 200000-BULK_PUT) (max_inflight_rpcs = 1000):

Master Endpoint 2:0
-------------------
	RPC Bandwidth (MB/sec): 112.66
	RPC Throughput (RPCs/sec): 588
	RPC Latencies (us):
		Min    : 5340
		25th  %: 442729
		Median : 1221371
		75th  %: 2936906
		Max    : 3502405
		Average: 1681142
		Std Dev: 1308472.80
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 3006315
		0:1 - 2913808
		1:0 - 434763
		1:1 - 465469

##################################################
Results for message size (1000-IOV 1000-IOV) (max_inflight_rpcs = 1000):

Master Endpoint 2:0
-------------------
	RPC Bandwidth (MB/sec): 80.71
	RPC Throughput (RPCs/sec): 42315
	RPC Latencies (us):
		Min    : 1187
		25th  %: 20187
		Median : 23322
		75th  %: 26833
		Max    : 30246
		Average: 23319
		Std Dev: 4339.87
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 26828
		0:1 - 26839
		1:0 - 20275
		1:1 - 20306

##################################################
Results for message size (1000-IOV 0-EMPTY) (max_inflight_rpcs = 1000):

Master Endpoint 2:0
-------------------
	RPC Bandwidth (MB/sec): 42.68
	RPC Throughput (RPCs/sec): 44758
	RPC Latencies (us):
		Min    : 935
		25th  %: 15880
		Median : 21444
		75th  %: 28434
		Max    : 34551
		Average: 22035
		Std Dev: 7234.26
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 28418
		0:1 - 28449
		1:0 - 16301
		1:1 - 16318

##################################################
Results for message size (0-EMPTY 1000-IOV) (max_inflight_rpcs = 1000):

Master Endpoint 2:0
-------------------
	RPC Bandwidth (MB/sec): 42.91
	RPC Throughput (RPCs/sec): 44991
	RPC Latencies (us):
		Min    : 789
		25th  %: 20224
		Median : 22195
		75th  %: 24001
		Max    : 26270
		Average: 21943
		Std Dev: 3039.50
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 24017
		0:1 - 23987
		1:0 - 20279
		1:1 - 20309

##################################################
Results for message size (0-EMPTY 0-EMPTY) (max_inflight_rpcs = 1000):

Master Endpoint 2:0
-------------------
	RPC Bandwidth (MB/sec): 0.00
	RPC Throughput (RPCs/sec): 47807
	RPC Latencies (us):
		Min    : 774
		25th  %: 16161
		Median : 20419
		75th  %: 25102
		Max    : 29799
		Average: 20633
		Std Dev: 5401.96
	RPC Failures: 0

	Endpoint results (rank:tag - Median Latency (us)):
		0:0 - 25103
		0:1 - 25099
		1:0 - 16401
		1:1 - 16421

Run daos_perf

On the client run daos_perf. Requires openmpi3.

(No need to create pool. Daos_perf will take care of it.)

Load mpi or set it in your environment.

# load openmpi module or set it's path in your environment
module load mpi/openmpi3-x86_64
or
export LD_LIBRARY_PATH=<openmpi lib path>:$LD_LIBRARY_PATH
export PATH=<openmpi bin path>:$PATH



Single Process daos_perf run example:

export D_LOG_FILE=/tmp/daos_perf_daos.log
daos_perf -g /etc/daos/daos_control.yml -o 10 -d 10 -a 10 -C 16 -A -R 'U;p F;i=5;p V' -P 1G -N 10G
   'U' means UPDATE, 'p' means outputing performance
   'F' means FETCH, parameter
   "i=5" indicates the FETCH test will repeat 5 times.
   'V' means verify correctness of data
   ‘o’ Number of Objects
   ‘d’ Number of dkeys
   ‘a’ Number of akeys
   ‘A’ Use Array value of akey. Single value is selected by default.
   ‘P’ Scm Pool size
   ‘N’ Nvme Pool Size

Test :
	DAOS LARGE (full stack, non-replica)
Pool :
	79cae139-272e-4ada-922e-98cc968126bf
Parameters :
	pool size     : SCM: 1024 MB, NVMe: 10240 MB
	credits       : 16 (sync I/O for -ve)
	obj_per_cont  : 10 x 1 (procs)
	dkey_per_obj  : 10
	akey_per_dkey : 10
	recx_per_akey : 16
	value type    : array
	stride size   : 64
	zero copy     : no
	VOS file      : <NULL>
Running test=UPDATE
Running UPDATE test (iteration=1)
UPDATE successfully completed:
	duration : 0.225937   sec
	bandwith : 4.322      MB/sec
	rate     : 70816.20   IO/sec
	latency  : 14.121     us (nonsense if credits > 1)
Duration across processes:
	MAX duration : 0.225937   sec
	MIN duration : 0.225937   sec
	Average duration : 0.225937   sec
Completed test=UPDATE
Running test=FETCH
Running FETCH test (iteration=5)
FETCH successfully completed:
	duration : 1.190614   sec
	bandwith : 4.101      MB/sec
	rate     : 67192.22   IO/sec
	latency  : 14.883     us (nonsense if credits > 1)
Duration across processes:
	MAX duration : 1.190614   sec
	MIN duration : 1.190614   sec
	Average duration : 1.190614   sec


Multi-process daos_perf example:

/usr/lib64/openmpi3/bin/orterun --enable-recovery --hostfile /path/to/hostfile_clients --map-by node --mca btl_openib_warn_default_gid_prefix "0" --mca btl "tcp,self" --mca oob "tcp" --mca pml "ob1" --mca btl_tcp_if_include "eth0" --np 4 --tag-output daos_perf -g /etc/daos/daos.yml -o 10 -d 10 -a 10 -C 16 -A -R 'U;p F;i=5;p V' -P 1G -N 10G
[1,0]<stdout>:Test :
[1,0]<stdout>:	DAOS LARGE (full stack, non-replica)
[1,0]<stdout>:Pool :
[1,0]<stdout>:	8ce8ae49-57e2-4d6e-b497-b50bbd6375dd
[1,0]<stdout>:Parameters :
[1,0]<stdout>:	pool size     : SCM: 1024 MB, NVMe: 10240 MB
[1,0]<stdout>:	credits       : 16 (sync I/O for -ve)
[1,0]<stdout>:	obj_per_cont  : 10 x 4 (procs)
[1,0]<stdout>:	dkey_per_obj  : 10
[1,0]<stdout>:	akey_per_dkey : 10
[1,0]<stdout>:	recx_per_akey : 16
[1,0]<stdout>:	value type    : array
[1,0]<stdout>:	stride size   : 64
[1,0]<stdout>:	zero copy     : no
[1,0]<stdout>:	VOS file      : <NULL>
[1,0]<stdout>:Running test=UPDATE
[1,0]<stdout>:Running UPDATE test (iteration=1)
[1,2]<stdout>:Running test=UPDATE
[1,2]<stdout>:Running UPDATE test (iteration=1)
[1,1]<stdout>:Running test=UPDATE
[1,1]<stdout>:Running UPDATE test (iteration=1)
[1,3]<stdout>:Running test=UPDATE
[1,3]<stdout>:Running UPDATE test (iteration=1)
[1,2]<stdout>:Completed test=UPDATE
[1,2]<stdout>:Running test=FETCH
[1,2]<stdout>:Running FETCH test (iteration=5)
[1,0]<stdout>:UPDATE successfully completed:
[1,1]<stdout>:Completed test=UPDATE
[1,1]<stdout>:Running test=FETCH
[1,0]<stdout>:	duration : 64.994374  sec
[1,0]<stdout>:	bandwith : 0.060      MB/sec
[1,0]<stdout>:	rate     : 984.70     IO/sec
[1,0]<stdout>:	latency  : 22.106     us (nonsense if credits > 1)
[1,0]<stdout>:Duration across processes:
[1,0]<stdout>:	MAX duration : 1.414762   sec
[1,0]<stdout>:	MIN duration : 1.064101   sec
[1,3]<stdout>:Completed test=UPDATE
[1,3]<stdout>:Running test=FETCH
[1,3]<stdout>:Running FETCH test (iteration=5)
[1,1]<stdout>:Running FETCH test (iteration=5)
[1,0]<stdout>:	Average duration : 1.194978   sec
[1,0]<stdout>:Completed test=UPDATE
[1,0]<stdout>:Running test=FETCH
[1,0]<stdout>:Running FETCH test (iteration=5)
[1,2]<stdout>:Completed test=FETCH
[1,2]<stdout>:Running test=VERIFY
[1,2]<stdout>:Running VERIFY test (iteration=1)
[1,0]<stdout>:FETCH successfully completed:
[1,0]<stdout>:	duration : 68.807437  sec
[1,0]<stdout>:	bandwith : 0.284      MB/sec
[1,0]<stdout>:	rate     : 4650.66    IO/sec
[1,0]<stdout>:	latency  : 16.333     us (nonsense if credits > 1)
[1,0]<stdout>:Duration across processes:
[1,0]<stdout>:	MAX duration : 5.226541   sec
[1,0]<stdout>:	MIN duration : 3.616002   sec
[1,0]<stdout>:	Average duration : 4.349847   sec
[1,0]<stdout>:Completed test=FETCH
[1,0]<stdout>:Running test=VERIFY
[1,0]<stdout>:Running VERIFY test (iteration=1)
[1,3]<stdout>:Completed test=FETCH
[1,3]<stdout>:Running test=VERIFY
[1,1]<stdout>:Completed test=FETCH
[1,1]<stdout>:Running test=VERIFY
[1,1]<stdout>:Running VERIFY test (iteration=1)
[1,3]<stdout>:Running VERIFY test (iteration=1)
[1,0]<stdout>:Completed test=VERIFY
[1,2]<stdout>:Completed test=VERIFY
[1,1]<stdout>:Completed test=VERIFY
[1,3]<stdout>:Completed test=VERIFY

Run daos_racer

On the client run daos_racer. Requires openmpi3.

(No need to create pool. Daos_racer will take care of it.)

Steps involve loading of openmpi or setting in environment if not already done. Set log file location and log level, and finally run.

# load openmpi module or set it's path in your environment
module load mpi/openmpi3-x86_64
or
export LD_LIBRARY_PATH=<openmpi lib path>:$LD_LIBRARY_PATH
export PATH=<openmpi bin path>:$PATH

# set environment var 
export D_LOG_FILE=/tmp/daos_racer_log
export D_LOG_MASK=ERR
 
# run daos_racer
/usr/bin/daos_racer -n /etc/daos/daos_control.yml -t 60

Run daos_test

On the client run daos_test. Requires openmpi3.

(No need to create pool. Daos_test will take care of it.)

Steps involve loading of openmpi or setting in environment if not already done. Set pool size environment variables, and finally run desired or all daos_tests.

# load openmpi module or set it's path in your environment
module load mpi/openmpi3-x86_64
or
export LD_LIBRARY_PATH=<openmpi lib path>:$LD_LIBRARY_PATH
export PATH=<openmpi bin path>:$PATH
 
export POOL_SCM_SIZE=8G
export POOL_NVME_SIZE=16G

(Example of Management Test)
/usr/lib64/openmpi3/bin/orterun --mca btl_openib_warn_default_gid_prefix 0 --mca pml ob1 --mca btl tcp,self --mca oob tcp --mca btl_tcp_if_include eth0 -n 1 --hostfile /path/to/hostfile_clients -x D_LOG_FILE=/tmp/daos_test.log --map-by node -x D_LOG_MASK=DEBUG -x DD_MASK=mgmt,io,md,epc,rebuild -x COVFILE=/tmp/test.cov /usr/bin/daos_test -n /etc/daos/daos_control.yml -m
=================
DAOS management tests..
=====================
[==========] Running 5 test(s).
[ RUN ] MGMT1: create/destroy pool on all tgts
creating pool synchronously ... success uuid = 6e3dbfee-ee2e-47ec-8b3c-6a42b62d7573
destroying pool synchronously ... success
[ OK ] MGMT1: create/destroy pool on all tgts
[ RUN ] MGMT2: create/destroy pool on all tgts (async)
async not supported[ OK ] MGMT2: create/destroy pool on all tgts (async)
[ RUN ] MGMT3: list-pools with no pools in sys
success t0: output npools=0
verifying pools[0..10], nfilled=0
success t1: pools[] over-sized
success t2: npools=0, non-NULL pools[] rc=0
success t3: in &npools NULL, -DER_INVAL
success
[ OK ] MGMT3: list-pools with no pools in sys
[ RUN ] MGMT4: list-pools with multiple pools in sys
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool 24aea230-2366-45a6-90e8-63ab36e58f8b
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool cb936b0c-c32f-4465-86f9-d18aa94a5338
setup: creating pool, SCM size=8 GB, NVMe size=16 GB
setup: created pool 02f0d551-843d-4677-9905-696205c1fb5c
success t0: output npools=4
verifying pools[0..14], nfilled=4
pool 24aea230-2366-45a6-90e8-63ab36e58f8b found in list result
pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e found in list result
pool cb936b0c-c32f-4465-86f9-d18aa94a5338 found in list result
pool 02f0d551-843d-4677-9905-696205c1fb5c found in list result
success t1: pools[] over-sized
success t2: npools=0, non-NULL pools[] rc=-1016
success t3: in &npools NULL, -DER_INVAL
verifying pools[0..4], nfilled=4
pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e found in list result
pool cb936b0c-c32f-4465-86f9-d18aa94a5338 found in list result
pool 02f0d551-843d-4677-9905-696205c1fb5c found in list result
pool 24aea230-2366-45a6-90e8-63ab36e58f8b found in list result
success t4: pools[] exact length
verifying pools[0..3], nfilled=0
success t5: pools[] under-sized
success
teardown: destroyed pool 24aea230-2366-45a6-90e8-63ab36e58f8b
teardown: destroyed pool 37dfd309-2f3e-48ab-8b9f-551d5a8cb41e
teardown: destroyed pool cb936b0c-c32f-4465-86f9-d18aa94a5338
teardown: destroyed pool 02f0d551-843d-4677-9905-696205c1fb5c
[ OK ] MGMT4: list-pools with multiple pools in sys
[ RUN ] MGMT5: retry MGMT_POOL_{CREATE,DESETROY} upon errors
setting DAOS_POOL_CREATE_FAIL_CORPC ... success
creating pool synchronously ... success uuid = a93bd58f-1c9a-4adf-8a96-e366e6f25c6a
destroying pool synchronously ... success
[ OK ] MGMT5: retry MGMT_POOL_{CREATE,DESETROY} upon errors
[==========] 5 test(s) run.
[ PASSED ] 5 test(s).

============ Summary src/tests/suite/daos_test.c
OK - NO TEST FAILURES
All DAOS TESTS
=============================
Tests: Use one of these arg(s) for specific test
daos_test -m|--mgmt
daos_test -p|--daos_pool_tests
daos_test -c|--daos_container_tests
daos_test -C|--capa
daos_test -U|--dedup
daos_test -z|--checksum
daos_test -Z|--ec_aggregation
daos_test -t|--base_tx
daos_test -T|--dist_tx
daos_test -i|--daos_io_tests
daos_test -x|--epoch_io
daos_test -A|--array
daos_test -D|--daos_array
daos_test -K|--daos_kv
daos_test -d|--degraded
daos_test -e|--daos_epoch_tests
daos_test -o|--daos_epoch_recovery_tests
daos_test -V|--verify_consistency
daos_test -R|--MD_replication_tests
daos_test -O|--oid_alloc
daos_test -r|--rebuild
daos_test -v|--rebuild_simple
daos_test -S|--rebuild_ec
daos_test -X|--degrade_ec
daos_test -b|--drain_simple
daos_test -B|--extend_simple
daos_test -N|--nvme_recovery
daos_test -a|--daos_all_tests
Default <daos_tests> runs all tests 


Create pool, posix container and then mount the container via fuse

Steps:

Create a pool.

Create a posix type container.

Make directory for dfuse mount point.

Mount dfuse.

Verify dfuse got mounted.

# create pool
dmg pool create --label=pool_2 --size=800G
Creating DAOS pool with automatic storage allocation: 800 GB NVMe + 6.00% SCM
Pool created with 6.00% SCM/NVMe ratio
---------------------------------------
  UUID          : b22220ea-740d-46bc-84ad-35ed3a28aa31
  Service Ranks : [1-3]                               
  Storage Ranks : [0-7]                               
  Total Size    : 848 GB                              
  SCM           : 48 GB (6.0 GB / rank)               
  NVMe          : 800 GB (100 GB / rank) 

# List pool
dmg pool list
Pool UUID                            Svc Replicas 
---------                            ------------ 
b22220ea-740d-46bc-84ad-35ed3a28aa31 [1-3]

# assign pool uuid to DAOS_POOL variable
export DAOS_POOL=b22220ea-740d-46bc-84ad-35ed3a28aa31

# Pool Query
dmg pool query $DAOS_POOL
Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
  Total size: 48 GB
  Free: 48 GB, min:750 MB, max:750 MB, mean:750 MB
- NVMe:
  Total size: 800 GB
  Free: 660 GB, min:10 GB, max:10 GB, mean:10 GB
Rebuild idle, 0 objs, 0 recs

# create container of type POSIX
daos container create --pool=$DAOS_POOL --type=POSIX
Successfully created container cd46cf6e-f886-4682-8077-e3cbcd09b43a


# assign container uuid to DAOS_CONT variable
export DAOS_CONT=cd46cf6e-f886-4682-8077-e3cbcd09b43a

# create dir for dfuse mountpoint on all client nodes
pdsh -w $CLIENT_NODES 'mkdir /tmp/daos_dfuse'

# mount dfuse
pdsh -w $CLIENT_NODES dfuse --pool=$DAOS_POOL --cont=$DAOS_CONT --mountpoint=/tmp/daos_dfuse/

#verify dfuse got mounted.
pdsh -w $CLIENT_NODES df /tmp/daos_dfuse

wolf-86: Filesystem     1K-blocks      Used Available Use% Mounted on
wolf-86: dfuse          828125000 136841868 691283133  17% /tmp/daos_dfuse
wolf-88: Filesystem     1K-blocks      Used Available Use% Mounted on
wolf-88: dfuse          828125000 136841868 691283133  17% /tmp/daos_dfuse
wolf-87: Filesystem     1K-blocks      Used Available Use% Mounted on
wolf-87: dfuse          828125000 136841868 691283133  17% /tmp/daos_dfuse

Run IOR

Use IOR to generate an HPC type POSIX I/O load to the POSIX container.

Build ior

git clone https://github.com/hpc/ior.git
cd ior
./bootstrap
mkdir build;cd build
../configure --with-daos=/usr --prefix=<your dir>
make
make install

Add <your dir>/lib to LD_LIBRARY_PATh and <your dir>/bin to PATH

Or

On all client nodes:

sudo yum install ior

Use ior to write and read around 150G of data

# load mpich module or set it's path in your environment
module load mpi/mpich-x86_64
or
export LD_LIBRARY_PATH=<mpich lib path>:$LD_LIBRARY_PATH
export PATH=<mpich bin path>:$PATH  


mpirun -hostfile /path/to/hostfile_clients -np 30 ior -a POSIX -b 5G -t 1M -v -W -w -r -R -i 1 -k -o /tmp/daos_dfuse/testfile

IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Apr 29 23:23:09 2021
Command line        : ior -a POSIX -b 5G -t 1M -v -W -w -r -R -i 1 -k -o /tmp/daos_dfuse/testfile
Machine             : Linux wolf-86.wolf.hpdd.intel.com
Start time skew across all tasks: 0.00 sec
TestID              : 0
StartTime           : Thu Apr 29 23:23:09 2021
Path                : /tmp/daos_dfuse/testfile
FS                  : 789.8 GiB   Used FS: 16.5%   Inodes: -0.0 Mi   Used Inodes: 0.0%
Participating tasks : 30

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /tmp/daos_dfuse/testfile
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : no tasks offsets
nodes               : 3
tasks               : 30
clients per node    : 10
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 5 GiB
aggregate filesize  : 150 GiB
verbose             : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Thu Apr 29 23:23:09 2021
write     1299.23    1299.84    0.022917    5242880    1024.00    10.79      118.17     0.000377   118.22     0   
Verifying contents of the file(s) just written.
Thu Apr 29 23:25:07 2021

Commencing read performance test: Thu Apr 29 23:25:35 2021

read      5429       5431       0.005523    5242880    1024.00    0.012188   28.28      0.000251   28.29      0   
Max Write: 1299.23 MiB/sec (1362.35 MB/sec)
Max Read:  5429.38 MiB/sec (5693.11 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        1299.23    1299.23    1299.23       0.00    1299.23    1299.23    1299.23       0.00  118.22343         NA            NA     0     30  10    1   0     0        1         0    0      1 5368709120  1048576  153600.0 POSIX      0
read         5429.38    5429.38    5429.38       0.00    5429.38    5429.38    5429.38       0.00   28.29054         NA            NA     0     30  10    1   0     0        1         0    0      1 5368709120  1048576  153600.0 POSIX      0
Finished            : Thu Apr 29 23:26:03 2021

Run mdtest

Create a substantial directory structure.

Use mdtest to create 30K files

 mpirun -hostfile /path/to/hostfile_clients -np 10 mdtest -a POSIX -z 0 -F -C -i 1 -n 3334 -e 4096 -d /tmp/daos_dfuse/ -w 4096

-- started at 04/29/2021 23:28:11 --

mdtest-3.4.0+dev was launched with 10 total task(s) on 3 node(s)
Command line used: mdtest '-a' 'POSIX' '-z' '0' '-F' '-C' '-i' '1' '-n' '3334' '-e' '4096' '-d' '/tmp/daos_dfuse/' '-w' '4096'
Path: /tmp/daos_dfuse
FS: 36.5 GiB   Used FS: 18.8%   Inodes: 2.3 Mi   Used Inodes: 5.9%

Nodemap: 1001001001
10 tasks, 33340 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       2943.697       2943.674       2943.686          0.006
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :       1079.858       1079.858       1079.858          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 04/29/2021 23:28:22 --

Run dbench

Install dbench on all client nodes:

sudo yum install dbench

From one of the client node:

dbench --clients-per-process 10 --directory /tmp/daos_dfuse/ --loadfile /usr/share/dbench/client.txt --timelimit 10 10

dbench version 4.00 - Copyright Andrew Tridgell 1999-2004

Running for 10 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 2 secs
failed to create barrier semaphore 
9 of 10 processes prepared for launch   0 sec
10 of 10 processes prepared for launch   0 sec
releasing clients
   0         3     0.00 MB/sec  warmup   1 sec  latency 826.199 ms
   0         7     0.00 MB/sec  warmup   2 sec  latency 269.284 ms
 100       114   288.13 MB/sec  execute   1 sec  latency 230.428 ms
 100       141   184.47 MB/sec  execute   2 sec  latency 246.159 ms
 100       166   147.88 MB/sec  execute   3 sec  latency 266.298 ms
 100       194   133.59 MB/sec  execute   4 sec  latency 255.767 ms
 100       219   121.64 MB/sec  execute   5 sec  latency 257.980 ms
 100       248   117.41 MB/sec  execute   6 sec  latency 278.191 ms
 100       274   112.64 MB/sec  execute   7 sec  latency 283.694 ms
 100       299   107.89 MB/sec  execute   8 sec  latency 274.483 ms
 100       325   104.57 MB/sec  execute   9 sec  latency 285.639 ms
 100  cleanup  10 sec
 100  cleanup  11 sec
  90  cleanup  12 sec
  70  cleanup  13 sec
  50  cleanup  14 sec
  35  cleanup  15 sec
  20  cleanup  16 sec
   0  cleanup  17 sec

 Operation      Count    AvgLat    MaxLat
 ----------------------------------------
 NTCreateX       3877    24.215   170.761
 Close           3800     0.004     0.022
 Qfileinfo       3110     1.488     4.579
 WriteX         18750     0.274     6.484

Throughput 104.568 MB/sec  100 clients  10 procs  max_latency=285.639 ms


List the dfuse mount point:

# 'testfile' comes from ior run
# 'test-dir.0-0' comes from mdtest run
# 'clients' comes from dbench run
ls /tmp/daos_dfuse
clients  test-dir.0-0  testfile

DataMover

DataMover using ‘daos’ utility (Single process)

Create Second container

# Create Second container
daos container create --pool $DAOS_POOL --type POSIX
Successfully created container 158469db-70d2-4a5d-aac9-3c06cbfa7459

export DAOS_CONT2=<cont uuid>


Pool Query before copy:

dmg -o /etc/daos/daos_control.yml pool query --pool $DAOS_POOL

Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
  Total size: 48 GB
  Free: 48 GB, min:743 MB, max:744 MB, mean:744 MB
- NVMe:
  Total size: 800 GB
  Free: 499 GB, min:7.7 GB, max:7.9 GB, mean:7.8 GB
Rebuild idle, 0 objs, 0 recs


Move data from POSIX directory into a DAOS container:

Daos 1.2 only supports directory copy if using 'daos filesystem copy'

# moving everything under /tmp/daos_dfuse to new cont $DAOS_CONT2
daos filesystem copy --src /tmp/daos_dfuse/ --dst daos://$DAOS_POOL/$DAOS_CONT2
Successfully copied to DAOS: /


Pool Query to confirm data got copied (Free space has reduced from last pool query):

dmg pool query $DAOS_POOL
Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
  Total size: 48 GB
  Free: 47 GB, min:738 MB, max:739 MB, mean:739 MB
- NVMe:
  Total size: 800 GB
  Free: 338 GB, min:5.1 GB, max:5.5 GB, mean:5.3 GB
Rebuild idle, 0 objs, 0 recs


Move data from DAOS container to POSIX directory:

mkdir /tmp/daos_dfuse/daos_container_copy

daos filesystem copy --src daos://$DAOS_POOL/$DAOS_CONT2 --dst /tmp/daos_dfuse/daos_container_copy

mkdir /tmp/daos_dfuse/daos_container_copy
mkdir: cannot create directory ‘/tmp/daos_dfuse/daos_container_copy’: File exists


Pool Query to confirm data got copied:

dmg pool query $DAOS_POOL
Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
  Total size: 48 GB
  Free: 47 GB, min:732 MB, max:733 MB, mean:732 MB
- NVMe:
  Total size: 800 GB
  Free: 128 GB, min:1.8 GB, max:2.3 GB, mean:2.0 GB
Rebuild idle, 0 objs, 0 recs


Check data inside the POSIX directories:

ls -latr /tmp/daos_dfuse/
total 157286400
-rw-rw-r-- 1 standan standan 161061273600 Apr 29 23:23 testfile
drwxrwxr-x 1 standan standan           64 Apr 29 23:28 test-dir.0-0
drwxrwxr-x 1 standan standan           64 Apr 29 23:30 clients
drwxrwxr-x 1 standan standan           64 Apr 30 00:02 daos_container_copy

ls -latr /tmp/daos_dfuse/daos_container_copy
drwx------ 1 standan standan 64 Apr 30 00:02 daos_dfuse
drwx------ 1 standan standan 64 Apr 30 00:11 testfile

DataMover using mpifileutils (allows you to move data in multi-process mode)

Build mpifileutils package

# load mpich module or set it's path in your environment
module load mpi/mpich-x86_64
or
export LD_LIBRARY_PATH=<mpich lib path>:$LD_LIBRARY_PATH
export PATH=<mpich bin path>:$PATH

# install daos-devel, if missing
sudo yum install -y daos-devel

# Build Dependencies
mkdir install
installdir=`pwd`/install

mkdir deps
cd deps
  wget https://github.com/hpc/libcircle/releases/download/v0.3/libcircle-0.3.0.tar.gz
  wget https://github.com/llnl/lwgrp/releases/download/v1.0.2/lwgrp-1.0.2.tar.gz
  wget https://github.com/llnl/dtcmp/releases/download/v1.1.0/dtcmp-1.1.0.tar.gz

  tar -zxf libcircle-0.3.0.tar.gz
  cd libcircle-0.3.0
    ./configure --prefix=$installdir
    make install
  cd ..

  tar -zxf lwgrp-1.0.2.tar.gz
  cd lwgrp-1.0.2
    ./configure --prefix=$installdir
    make install
  cd ..

  tar -zxf dtcmp-1.1.0.tar.gz
  cd dtcmp-1.1.0
    ./configure --prefix=$installdir --with-lwgrp=$installdir
    make install
  cd ..
cd ..


# Build mpifileutils
git clone https://github.com/hpc/mpifileutils
mkdir build
cd build

cmake3 ../mpifileutils/ -DWITH_DTCMP_PREFIX=<path/to/dtcmp/install> -DWITH_LibCircle_PREFIX=<path/to/lib/circle/install> -DWITH_CART_PREFIX=/usr/ -DWITH_DAOS_PREFIX=/usr/ -DCMAKE_INSTALL_PREFIX=<path/where/mpifileutils/need/to/be/installed> -DENABLE_DAOS=ON -DENABLE_LIBARCHIVE=OFF

make install

# On launch node set mpifileutils LD_LIBRARY_PATH and PATH
export LD_LIBRARY_PATH=<mpifileutils/lib/path>:$LD_LIBRARY_PATH
export PATH=<mpifileutils/bin/path>:$PATH  


 Create Second container

daos container create --pool $DAOS_POOL --type POSIX
Successfully created container caf0135c-def8-45a5-bac3-d0b969e67c8b

export DAOS_CONT2=<cont uuid>


Move data from POSIX directory into a DAOS container:

mpirun -hostfile /path/to/hostfile -np 16 /path/to/mpifileutils/install/bin/dcp --bufsize 64MB --chunksize 128MB /tmp/daos_dfuse daos://$DAOS_POOL/$DAOS_CONT2/

[2021-04-30T01:16:48] Walking /tmp/daos_dfuse
[2021-04-30T01:16:58] Walked 24207 items in 10.030 secs (2413.415 items/sec) ...
[2021-04-30T01:17:01] Walked 34245 items in 13.298 secs (2575.138 items/sec) ...
[2021-04-30T01:17:01] Walked 34245 items in 13.300 seconds (2574.867 items/sec)
[2021-04-30T01:17:01] Copying to /
[2021-04-30T01:17:01] Items: 34245
[2021-04-30T01:17:01]   Directories: 904
[2021-04-30T01:17:01]   Files: 33341
[2021-04-30T01:17:01]   Links: 0
[2021-04-30T01:17:01] Data: 150.127 GiB (4.611 MiB per file)
[2021-04-30T01:17:01] Creating 904 directories
[2021-04-30T01:17:01] Creating 33341 files.
[2021-04-30T01:17:02] Copying data.
[2021-04-30T01:17:12] Copied 4.049 GiB (3%) in 10.395 secs (398.867 MiB/s) 375 secs left ...
[2021-04-30T01:22:37] Copied 8.561 GiB (6%) in 335.113 secs (26.160 MiB/s) 5541 secs left ...
[2021-04-30T01:22:37] Copied 150.127 GiB (100%) in 335.113 secs (458.742 MiB/s) done
[2021-04-30T01:22:37] Copy data: 150.127 GiB (161197834240 bytes)
[2021-04-30T01:22:37] Copy rate: 458.741 MiB/s (161197834240 bytes in 335.114 seconds)
[2021-04-30T01:22:37] Syncing data to disk.
[2021-04-30T01:22:37] Sync completed in 0.017 seconds.
[2021-04-30T01:22:37] Fixing permissions.
[2021-04-30T01:22:37] Updated 34245 items in 0.176 seconds (194912.821 items/sec)
[2021-04-30T01:22:37] Syncing directory updates to disk.
[2021-04-30T01:22:37] Sync completed in 0.012 seconds.
[2021-04-30T01:22:37] Started: Apr-30-2021,01:17:01
[2021-04-30T01:22:37] Completed: Apr-30-2021,01:22:37
[2021-04-30T01:22:37] Seconds: 336.013
[2021-04-30T01:22:37] Items: 34245
[2021-04-30T01:22:37]   Directories: 904
[2021-04-30T01:22:37]   Files: 33341
[2021-04-30T01:22:37]   Links: 0
[2021-04-30T01:22:37] Data: 150.127 GiB (161197834240 bytes)
[2021-04-30T01:22:37] Rate: 457.513 MiB/s (161197834240 bytes in 336.013 seconds)


Pool Query to verify data got copied (free space should reduce):

dmg pool query --pool $DAOS_POOL

Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
  Total size: 48 GB
  Free: 47 GB, min:734 MB, max:735 MB, mean:735 MB
- NVMe:
  Total size: 800 GB
  Free: 338 GB, min:5.2 GB, max:5.4 GB, mean:5.3 GB
Rebuild idle, 0 objs, 0 recs


Move data from DAOS container to POSIX directory:

mkdir /tmp/daos_dfuse/daos_container_copy

mpirun -hostfile /path/to/hostfile -np 16 dcp --bufsize 64MB --chunksize 128MB daos://$DAOS_POOL/$DAOS_CONT2/ /tmp/daos_dfuse/daos_container_copy

[2021-04-30T01:26:11] Walking /
[2021-04-30T01:26:14] Walked 34246 items in 2.648 secs (12930.593 items/sec) ...
[2021-04-30T01:26:14] Walked 34246 items in 2.650 seconds (12923.056 items/sec)
[2021-04-30T01:26:14] Copying to /tmp/daos_dfuse/daos_container_copy
[2021-04-30T01:26:14] Items: 34246
[2021-04-30T01:26:14]   Directories: 905
[2021-04-30T01:26:14]   Files: 33341
[2021-04-30T01:26:14]   Links: 0
[2021-04-30T01:26:14] Data: 150.127 GiB (4.611 MiB per file)
[2021-04-30T01:26:14] Creating 905 directories
[2021-04-30T01:26:14] Original directory exists, skip the creation: `/tmp/daos_dfuse/daos_container_copy/' (errno=17 File exists)
[2021-04-30T01:26:14] Creating 33341 files.
[2021-04-30T01:26:19] Copying data.
[2021-04-30T01:26:29] Copied 3.819 GiB (3%) in 10.213 secs (382.922 MiB/s) 391 secs left ...
[2021-04-30T01:32:02] Copied 150.127 GiB (100%) in 343.861 secs (447.070 MiB/s) done
[2021-04-30T01:32:02] Copy data: 150.127 GiB (161197834240 bytes)
[2021-04-30T01:32:02] Copy rate: 447.069 MiB/s (161197834240 bytes in 343.862 seconds)
[2021-04-30T01:32:02] Syncing data to disk.
[2021-04-30T01:32:02] Sync completed in 0.020 seconds.
[2021-04-30T01:32:02] Fixing permissions.
[2021-04-30T01:32:17] Updated 34162 items in 14.955 secs (2284.295 items/sec) ...
[2021-04-30T01:32:17] Updated 34246 items in 14.955 secs (2289.890 items/sec) done
[2021-04-30T01:32:17] Updated 34246 items in 14.956 seconds (2289.772 items/sec)
[2021-04-30T01:32:17] Syncing directory updates to disk.
[2021-04-30T01:32:17] Sync completed in 0.022 seconds.
[2021-04-30T01:32:17] Started: Apr-30-2021,01:26:14
[2021-04-30T01:32:17] Completed: Apr-30-2021,01:32:17
[2021-04-30T01:32:17] Seconds: 363.327
[2021-04-30T01:32:17] Items: 34246
[2021-04-30T01:32:17]   Directories: 905
[2021-04-30T01:32:17]   Files: 33341
[2021-04-30T01:32:17]   Links: 0
[2021-04-30T01:32:17] Data: 150.127 GiB (161197834240 bytes)
[2021-04-30T01:32:17] Rate: 423.118 MiB/s (161197834240 bytes in 363.327 seconds)


Pool Query to very data was copied:

dmg pool query --pool $DAOS_POOL

Pool b22220ea-740d-46bc-84ad-35ed3a28aa31, ntarget=64, disabled=0, leader=1, version=1
Pool space info:
- Target(VOS) count:64
- SCM:
  Total size: 48 GB
  Free: 47 GB, min:728 MB, max:730 MB, mean:729 MB
- NVMe:
  Total size: 800 GB
  Free: 176 GB, min:2.6 GB, max:3.0 GB, mean:2.8 GB
Rebuild idle, 0 objs, 0 recs


Check data inside the POSIX directories:

ls -latr /tmp/daos_dfuse/

total 157286400
-rw-rw-r-- 1 standan standan 161061273600 Apr 29 23:23 testfile
drwxrwxr-x 1 standan standan           64 Apr 29 23:28 test-dir.0-0
drwxrwxr-x 1 standan standan           64 Apr 29 23:30 clients
drwxr-xr-x 1 standan standan           64 Apr 30 01:25 daos_container_copy

ls -latr /tmp/daos_dfuse/daos_container_copy
drwxr-xr-x 1 standan standan 64 Apr 30 01:26 daos_dfuse

 For more details on datamover reference: https://github.com/hpc/mpifileutils/blob/master/DAOS-Support.md

Clean Up

Remove one of the copy created using datamover

rm -rf /tmp/daos_dfuse/daos_container_copy

Remove dfuse mountpoint:

# unmount dfuse
pdsh -w $CLIENT_NODES 'fusermount3 -uz /tmp/daos_dfuse'

# remove mount dir
pdsh -w $CLIENT_NODES rm -rf /tmp/daos_dfuse

List containers to be destroyed:

# list containers
daos pool list-containers --pool $DAOS_POOL  # sample output

# sample output
cd46cf6e-f886-4682-8077-e3cbcd09b43a
caf0135c-def8-45a5-bac3-d0b969e67c8b

Destroy Containers:

# destroy container1
daos container destroy --pool $DAOS_POOL --cont $DAOS_CONT

# destroy container2
daos container destroy --pool $DAOS_POOL --cont $DAOS_CONT2

List Pools to be destroyed:

# list pool
dmg pool list

# sample output
Pool UUID                            Svc Replicas 
---------                            ------------ 
b22220ea-740d-46bc-84ad-35ed3a28aa31 [1-3]

Destroy Pool:

# destroy pool
dmg pool destroy --pool $DAOS_POOL

Stop Agents:

# stop agents
pdsh -S -w $CLIENT_NODES "sudo systemctl stop daos_agent"

Stop Servers:

# stop servers
pdsh -S -w $SERVER_NODES "sudo systemctl stop daos_server"