For the near term, offloading manual testing from Liang such that tests that are not in CI yet are still run to give some basic level of product confidence, while freeing up Liang to focus on the myriad tasks he has on his plate.

daos_test:

______________________________________________

NOTE: In the daos_m repo, there's a script called launcher.sh that was added to run daos_test while also stopping/starting servers in between subtests that kill servers. This should probably be the de facto way to run daos_test going forward.

Run Server, N nodes:

At last check, 15 nodes are required for all subtests to run, as the rebuild tests will kill servers, and server rejoining server group is not supported today.

orterun --mca mtl ^psm2,ofi -N 1 --hostfile ~/hostlists/daos_server_hostlist --enable-recovery --report-uri ~/scripts/uri.txt daos_server -c 1 &

daos_test:

orterun --mca mtl ^psm2,ofi -np 1 --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daos_test -mpcCiAeoRd

CLEAN EVERYTHING

Relaunch server
Run rebuild tests, N nodes:
orterun --mca mtl ^psm2,ofi -np 1 --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daos_test -r

CLEAN EVERYTHING

Relaunch server
Run OID allocation tests, N nodes:
orterun --mca mtl ^psm2,ofi -np 1 --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daos_test -O

_______________________________________________

CLEAN EVERYTHING

IOR:

_______________________________________________

Set up envirables:
export LD_LIBRARY_PATH=/home/sdwillso/daos_m/opt/ofi/lib/:/home/sdwillso/daos_m/opt/cart/lib/:/home/sdwillso/daos_m/install/lib/:$LD_LIBRARY_PATH

Run server:

orterun --mca mtl ^psm2,ofi -N 1 --hostfile ~/hostlists/daos_server_hostlist --enable-recovery --report-uri ~/scripts/uri.txt daos_server -c 1 &

Create Pool:

orterun --mca mtl ^psm2,ofi -np 1 --ompi-server file:~/scripts/uri.txt dmg create --size=10G

Run IOR:

orterun -np 1 --hostfile ~/hostlists/daos_client_hostlist --mca mtl ^psm2,ofi  --ompi-server file:~/scripts/uri.txt ior -v -W -i 5 -a DAOS -w -o `uuidgen` -b 5g -t 1m -O daospool=$POOL,daosrecordsize=1m,daosstripesize=1m,daosstripecount=1024,daosaios=16,daosobjectclass=LARGE,daosPoolSvc=$POOL_LEADER,daosepoch=1

_______________________________________________

CLEAN EVERYTHING

daosbench:

_______________________________________________

Set environment for single record type implicit aggregation:
-x DAOS_IMPLICIT_PURGE=1 added to orterun server launch command, if desired

Run server:

orterun --mca mtl ^psm2,ofi -N 1 --hostfile ~/hostlists/daos_server_hostlist --enable-recovery --report-uri ~/scripts/uri.txt daos_server -c 1 &

Create Pool:

orterun --mca mtl ^psm2,ofi -np 1 --ompi-server file:~/scripts/uri.txt dmg create --size=4G

Run daos_bench:

orterun -np 1 --mca mtl ^psm2,ofi  --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-idx-update --testid=1 --svc=$POOL_LEADER --dpool=$POOL --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000

repeat with: kv-dkey-update, kv-akey-update, kv-dkey-fetch, kv-akey-fetch

_______________________________________________

CLEAN EVERYTHING

daos_perf:

_______________________________________________

Set environment for single record type implicit aggregation:
-x DAOS_IMPLICIT_PURGE=1 added to orterun server launch command, if desired

Run server:

orterun --mca mtl ^psm2,ofi -N 1 --hostfile ~/hostlists/daos_server_hostlist --enable-recovery --report-uri ~/scripts/uri.txt daos_server -c 1 &

Run daos_perf:

1K  records:

CREDITS=1 ./daos/daos_m/src/tests/daos_perf.sh daos 200 1000 1K

CREDITS=8 ./daos/daos_m/src/tests/daos_perf.sh daos 200 1000 1K

4k records

CREDITS=1 ./daos/daos_m/src/tests/daos_perf.sh daos 200 1000 4K

_______________________________________________

MPICH:

_______________________________________________

Building

Install autotools latest versions (autoconf, automake, libtool) or use Mohamad's:

export LD_LIBRARY_PATH=/home/mschaara/install/autotools/lib:$LD_LIBRARY_PATH

export PATH=/home/mschaara/install/autotools/bin:$PATH

Setup envirables:

export DAOS_DIR=/home/sdwillso/daos/daos_m/install/

export LD_LIBRARY_PATH=$DAOS_DIR/lib/:$DAOS_DIR/lib/daos_srv:$LD_LIBRARY_PATH

export PATH=$DAOS_DIR/bin:$PATH

export CPATH=$DAOS_DIR/include/:$CPATH

export C_INCLUDE_PATH=$DAOS_DIR/include:$C_INCLUDE_PATH

export MPI_LIB=""

Get MPICH with DAOS ADIO branch:

git clone https://github.com/mchaarawi/mpich.git

cd mpich

git checkout daos_adio

git submodule init

git submodule update

./autogen.sh

mkdir build

cd build

../configure --prefix=/home/sdwillso/mpich/install/ --enable-fortran=all --enable-romio --enable-cxx --enable-g=all --enable-debuginfo --with-file-system=ufs+daos --with-daos=/home/sdwillso/daos/daos_m/install/ --with-cart=/home/sdwillso/daos/daos_m/opt/cart/

make -j8

make install

IMPORTANT NOTE: must keep strict separation between server and client here, ie don't do "run server as background process and then run client right after on same node"

Set up client envirables:

export PATH=/home/sdwillso/mpich/install/bin:$PATH

export LD_LIBRARY_PATH=/home/sdwillso/mpich/install/lib:$LD_LIBRARY_PATH                                                                                                                                       

export DAOS_SINGLETON_CLI=1

export CRT_ATTACH_INFO_PATH=/home/sdwillso/tmp/

export FI_PSM2_DISCONNECT=1

export CRT_CTX_NUM=8

The last two are for PSM2, so you don’t need them if you are using sockets provider.

CRT_ATTACH_INFO_PATH should point to the same thing to what you pass the -a to for daos_server

Build ROMIO test suite:

On client:

cd mpich_directory/build/src/mpi/romio/test

make -j8

Run Tests

Set on Server:

export CRT_CTX_NUM=8

Run server:

orterun --mca mtl ^psm2,ofi -N 1 --hostfile ~/hostlists/daos_mpich_hostlist --enable-recovery --report-uri ~/scripts/uri.txt daos_server -a/home/sdwillso/tmp/ -c 1 &

Create a pool with dmg:

mpirun -np 1 dmg create --size=1G

And set env variables DAOS_POOL, DAOS_SVCL to the pool uuid and svcl rank list respectively.

On Boro there is a script that does that in /home/mschaara/create_pool.sh

Run:

./run_daos_tests daos:test_file

Note that the daos: prefix tells mpich to use the daos adio driver. If you don’t have that, it will use the regular posix driver, and you won’t be running with the DAOS driver.

_______________________________________________