Changes in this version from the one before (ISC22) include:
remove dependence on mpifileutils, and use a new pfind application.
DAOS - See https://docs.daos.io/latest/admin/hardware/ for installation and setup instructions
MPI - any version / implementation
clush - See https://clustershell.readthedocs.io/en/latest/install.html for installation
Alternatives are possible, though examples are not provided in these instructions.
These instructions assume the following paths. For simplicity, you can set these variables to the actual locations where you have/want these installed.
After setting these variables, most of the scripts can be "copy-pasted".
MY_DAOS_INSTALL_PATH=${HOME}/install/daos
MY_IO500_PATH=${HOME}/io500 |
git clone https://github.com/IO500/io500.git -b io500-sc22 "${MY_IO500_PATH}" &&
cd "${MY_IO500_PATH}" |
Point to the new pfind branch
Build ior with DFS support
Assuming MY_DAOS_INSTALL_PATH is set, you can run:
( Note: for the SC22 version of IO500, line 9 of this patch has been changed to the new SC22 version)
cat << EOF > io500_prepare.patch
diff --git a/prepare.sh b/prepare.sh
index e38cae6..54dbba5 100755
--- a/prepare.sh
+++ b/prepare.sh
@@ -8,7 +8,7 @@ echo It will output OK at the end if builds succeed
echo
IOR_HASH=06fc08e147600f4e5896a5b9b2bf8f1c4a79121f
-PFIND_HASH=62c3a7e31
+PFIND_HASH=dfs_find
INSTALL_DIR=\$PWD
BIN=\$INSTALL_DIR/bin
@@ -59,7 +59,7 @@ function get_ior {
function get_pfind {
echo "Preparing parallel find"
- git_co https://github.com/VI4IO/pfind.git pfind \$PFIND_HASH
+ git_co https://github.com/mchaarawi/pfind pfind \$PFIND_HASH
}
function get_schema_tools {
@@ -73,7 +73,7 @@ function build_ior {
pushd "\$BUILD"/ior
./bootstrap
# Add here extra flags
- ./configure --prefix="\$INSTALL_DIR"
+ ./configure --prefix="\$INSTALL_DIR" --with-daos=\${MY_DAOS_INSTALL_PATH}
cd src
\$MAKE clean
\$MAKE install
EOF
git apply io500_prepare.patch |
The Makefile needs to be updated to use the actual install location of DAOS. you can run:
cat << EOF > io500_Makefile.patch
diff --git a/Makefile b/Makefile
index ee5cee9..d8fc0e4 100644
--- a/Makefile
+++ b/Makefile
@@ -1,10 +1,12 @@
CC = mpicc
CFLAGS += -std=gnu99 -Wall -Wempty-body -Werror -Wstrict-prototypes -Werror=maybe-uninitialized -Warray-bounds
+CFLAGS += -I\${MY_DAOS_INSTALL_PATH}/include
IORCFLAGS = \$(shell grep CFLAGS ./build/ior/src/build.conf | cut -d "=" -f 2-)
CFLAGS += -g3 -lefence -I./include/ -I./src/ -I./build/pfind/src/ -I./build/ior/src/
IORLIBS = \$(shell grep LDFLAGS ./build/ior/src/build.conf | cut -d "=" -f 2-)
LDFLAGS += -lm \$(IORCFLAGS) \$(IORLIBS) # -lgpfs # may need some additional flags as provided to IOR
+LDFLAGS += -L\${MY_DAOS_INSTALL_PATH}/lib64 -ldaos -ldaos_common -ldfs -lgurt -luuid
VERSION_GIT=\$(shell git describe --always --abbrev=12)
VERSION_TREE=\$(shell git diff src | wc -l | sed -e 's/ *//g' -e 's/^0//' | sed "s/\([0-9]\)/-\1/")
EOF
git apply io500_Makefile.patch |
${MY_IO500_PATH}/prepare.sh |
A sample config-full.ini file for reference: https://github.com/mchaarawi/io500/blob/main/config-full-isc22.ini
If you want to download this:
wget https://raw.githubusercontent.com/mchaarawi/io500/ini_files/config-full-isc22.ini |
you need to change the result dir:
https://github.com/mchaarawi/io500/blob/ini_files/config-full-isc22.ini#L4
to point to a directory where the results will be stored. This directory is required to be accessible from rank 0 of the io-500 application. So it can be either:
A shared filesystem (example: an NFS, dfuse, lustre fs) accessible from the first node in the hostfile where rank 0 is running.
A local file system (/tmp/results) on the first node in the hostfile where rank 0 is running.
After the run is complete, the result files are all stored under this directory.
When running at first, set a short stonewall (5 seconds) to just verify everything runs fine.
For [find] the nprocs setting under that should be the same or less than the number of processes you want to run with the entire workflow (in io500.sh).
For documentation on creating pools, see https://docs.daos.io/latest/admin/pool_operations/.
For documentation on creating containers, see https://docs.daos.io/latest/user/container/.
For example:
dmg pool create -z 100G --label io500_pool daos container create --type POSIX --pool io500_pool --label=io500_cont |
export DAOS_POOL=io500_pool export DAOS_CONT=io500_cont |
Note that when using Intel MPI, some extra environment variables are required as detailed on:
https://docs.daos.io/v2.0/user/mpi-io/?h=intel+mpi#intel-mpi
This will replace $DAOS_POOL, $DAOS_CONT with their actual values.
envsubst < config-full-isc22.ini > temp.ini |
Run the binary directly with or without the extended mode:
mpirun -np 16 --hostfile ~/config/cli_hosts ./io500 temp.ini mpirun -np 16 --hostfile ~/config/cli_hosts ./io500 temp.ini --mode=extended |
The extended mode is not required for an official submission and will extend your runtime significantly. After the run completes, you will need to tar up the result dir for that run.
Note that some versions of OpenMPI require setting the environment variables on the mpirun command line, so one needs to add the environment variables that are mentioned above on the mpirun command line with the following format:
mpirun -np 16 -x env1=value1 -x env2=value2 --hostfile ~/config/cli_hosts ./io500 temp.ini |
Run the io-500.sh script:
This requires mounting dfuse on the launch node only (not all the compute nodes):
mkdir /tmp/dfuse dfuse --pool=$DAOS_POOL --container=$DAOS_CONT -m /tmp/dfuse/ |
Then, edit the io500.sh launch script with the mpirun command and change the local workdir, to add the dfuse prefix
diff --git a/io500.sh b/io500.sh
index 5f0bb89..dd62c35 100755
--- a/io500.sh
+++ b/io500.sh
@@ -12,10 +12,10 @@
# This script takes its parameters from the same .ini file as io500 binary.
io500_ini="$1" # You can set the ini file here
io500_mpirun="mpiexec"
-io500_mpiargs="-np 2"
+io500_mpiargs="-np 16 --hostfile ~/config/cli_hosts"
function setup(){
- local workdir="$1"
+ local workdir="/tmp/dfuse/$1"
local resultdir="$2"
mkdir -p $workdir $resultdir
|
Note that some versions of OpenMPI require setting the environment variables on the mpirun command line, so one needs to add the environment variables that are mentioned above on the mpirun command line in the script here with the following format:
io500_mpiargs="-np 16 -x env1=value1 -x env2=value2 --hostfile ~/config/cli_hosts |
Then run the io500.sh script which will tar the results for you at the end and place them in the result directory you specified in the ini file:
./io500.sh temp.ini |
Lastly umount dfuse on the launch node:
fusermount3 -u /tmp/dfuse/ |
The tarball generated at the end (whether ran the binary or with the script) with the results can be submitted to the io500 committee for consideration.