For the near term, offloading manual testing from Liang such that tests that are not in CI yet are still run to give some basic level of product confidence, while freeing up Liang to focus on the myriad tasks he has on his plate.
daos_test:
______________________________________________
Run Server, N nodes:
At last check, 15 nodes are required for all subtests to run, as the rebuild tests will kill servers, and server rejoining server group is not supported today.
orterun --mca mtl ^psm2,ofi -N 1 --hostfile ~/hostlists/daos_server_hostlist --enable-recovery --report-uri ~/scripts/uri.txt daos_server -c 1 &
daos_test:
orterun --mca mtl ^psm2,ofi -np 1 --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daos_test -mpcCiAeoRd
CLEAN EVERYTHING
Relaunch server
Run rebuild tests, N nodes:
orterun --mca mtl ^psm2,ofi -np 1 --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daos_test -r
CLEAN EVERYTHING
Relaunch server
Run OID allocation tests, N nodes:
orterun --mca mtl ^psm2,ofi -np 1 --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daos_test -O
_______________________________________________
CLEAN EVERYTHING
IOR:
_______________________________________________
Set up envirables:
export LD_LIBRARY_PATH=/home/sdwillso/daos_m/opt/ofi/lib/:/home/sdwillso/daos_m/opt/cart/lib/:/home/sdwillso/daos_m/install/lib/:$LD_LIBRARY_PATH
Run server:
orterun --mca mtl ^psm2,ofi -N 1 --hostfile ~/hostlists/daos_server_hostlist --enable-recovery --report-uri ~/scripts/uri.txt daos_server -c 1 &
Create Pool:
orterun --mca mtl ^psm2,ofi -np 1 --ompi-server file:~/scripts/uri.txt dmg create --size=10G
Run IOR:
orterun -np 1 --hostfile ~/hostlists/daos_client_hostlist --mca mtl ^psm2,ofi --ompi-server file:~/scripts/uri.txt ior -v -W -i 5 -a DAOS -w -o `uuidgen` -b 5g -t 1m -O daospool=$POOL,daosrecordsize=1m,daosstripesize=1m,daosstripecount=1024,daosaios=16,daosobjectclass=LARGE,daosPoolSvc=$POOL_LEADER,daosepoch=1
_______________________________________________
CLEAN EVERYTHING
daosbench:
_______________________________________________
Set environment for single record type implicit aggregation:
-x DAOS_IMPLICIT_PURGE=1 added to orterun server launch command, if desired
Run server:
orterun --mca mtl ^psm2,ofi -N 1 --hostfile ~/hostlists/daos_server_hostlist --enable-recovery --report-uri ~/scripts/uri.txt daos_server -c 1 &
Create Pool:
orterun --mca mtl ^psm2,ofi -np 1 --ompi-server file:~/scripts/uri.txt dmg create --size=4G
Run daos_bench:
orterun -np 1 --mca mtl ^psm2,ofi --hostfile ~/hostlists/daos_client_hostlist --ompi-server file:~/scripts/uri.txt daosbench --test=kv-idx-update --testid=1 --svc=$POOL_LEADER --dpool=$POOL --container=`uuidgen` --object-class=tiny --aios=32 --indexes=1000000
repeat with: kv-dkey-update, kv-akey-update, kv-dkey-fetch, kv-akey-fetch
_______________________________________________
CLEAN EVERYTHING
daos_perf:
_______________________________________________
Set environment for single record type implicit aggregation:
-x DAOS_IMPLICIT_PURGE=1 added to orterun server launch command, if desired
Run server:
orterun --mca mtl ^psm2,ofi -N 1 --hostfile ~/hostlists/daos_server_hostlist --enable-recovery --report-uri ~/scripts/uri.txt daos_server -c 1 &
Run daos_perf:
1K records:
CREDITS=1 ./daos/daos_m/src/tests/daos_perf.sh daos 200 1000 1K
CREDITS=8 ./daos/daos_m/src/tests/daos_perf.sh daos 200 1000 1K
4k records
CREDITS=1 ./daos/daos_m/src/tests/daos_perf.sh daos 200 1000 4K
_______________________________________________
MPICH:
_______________________________________________
Building
Install autotools latest versions (autoconf, automake, libtool) or use Mohamad's:
export LD_LIBRARY_PATH=/home/mschaara/install/autotools/lib:$LD_LIBRARY_PATH
export PATH=/home/mschaara/install/autotools/bin:$PATH
Setup envirables:
export DAOS_DIR=/home/sdwillso/daos/daos_m/install/
export LD_LIBRARY_PATH=$DAOS_DIR/lib/:$DAOS_DIR/lib/daos_srv:$LD_LIBRARY_PATH
export PATH=$DAOS_DIR/bin:$PATH
export CPATH=$DAOS_DIR/include/:$CPATH
export C_INCLUDE_PATH=$DAOS_DIR/include:$C_INCLUDE_PATH
export MPI_LIB=""
Get MPICH with DAOS ADIO branch:
git clone https://github.com/mchaarawi/mpich.git
cd mpich
git checkout daos_adio
git submodule init
git submodule update
./autogen.sh
mkdir build
cd build
../configure --prefix=/home/sdwillso/mpich/install/ --enable-fortran=all --enable-romio --enable-cxx --enable-g=all --enable-debuginfo --with-file-system=ufs+daos --with-daos=/home/sdwillso/daos/daos_m/install/ --with-cart=/home/sdwillso/daos/daos_m/opt/cart/
make -j8
make install
IMPORTANT NOTE: must keep strict separation between server and client here, ie don't do "run server as background process and then run client right after on same node"
Set up client envirables:
export PATH=/home/sdwillso/mpich/install/bin:$PATH
export LD_LIBRARY_PATH=/home/sdwillso/mpich/install/lib:$LD_LIBRARY_PATH
export DAOS_SINGLETON_CLI=1
export CRT_ATTACH_INFO_PATH=/home/sdwillso/tmp/
export FI_PSM2_DISCONNECT=1
export CRT_CTX_NUM=8
The last two are for PSM2, so you don’t need them if you are using sockets provider.
CRT_ATTACH_INFO_PATH should point to the same thing to what you pass the -a to for daos_server
Build ROMIO test suite:
On client:
cd mpich_directory/build/src/mpi/romio/test
make -j8
Run Tests
Set on Server:
export CRT_CTX_NUM=8
Run server:
orterun --mca mtl ^psm2,ofi -N 1 --hostfile ~/hostlists/daos_mpich_hostlist --enable-recovery --report-uri ~/scripts/uri.txt daos_server -a/home/sdwillso/tmp/ -c 1 &
Create a pool with dmg:
mpirun -np 1 dmg create --size=1G
And set env variables DAOS_POOL, DAOS_SVCL to the pool uuid and svcl rank list respectively.
On Boro there is a script that does that in /home/mschaara/create_pool.sh
Run:
./run_daos_tests daos:test_file
Note that the daos: prefix tells mpich to use the daos adio driver. If you don’t have that, it will use the regular posix driver, and you won’t be running with the DAOS driver.
_______________________________________________