Table of Contents
| Table of Contents | ||||
|---|---|---|---|---|
|
Notes
These instructions involve a new way to run io-500 with DAOS that does not require using dfuse on all client nodes; thus the instructions are a little different than previous versions.
More specifically, the changes are in the run section of the benchmark and the following changes should be noted vs previous versions:
mfu branch should be updated (still same branch name, just git pull, or fresh checkout).
io500 app is updated to sc21 tag.
should NOT export DAOS_FUSE env variable.
should export a new env variable on the client env - MFU_POSIX_TS=1 (can be done with the mpirun command)
this is an intermediate step to still support the older way of running and will not be required in future versions.
support for pool and container labels have been added (don't need to use uuids).
a new io-500 ini file that supports this new mode without dfuse, and also adds parameters for an extended io-500 run.
dfuse is still required only on the login node when running io-500 using the io-500.sh script
it is not required though if running the binary directly.
The run io-500 sections below has more details
Pre-requisites
DAOS - See https://docs.daos.io/latest/admin/hardware/ for installation and setup instructions
MPI - any version / implementation
cmake 3.1+
clush - See https://clustershell.readthedocs.io/en/latest/install.html for installation
Alternatives are possible, though examples are not provided in these instructions.
Build Paths
These instructions assume the following paths. For simplicity, you can set these variables to the actual locations where you have/want these installed.
...
| Code Block | ||
|---|---|---|
| ||
MY_DAOS_INSTALL_PATH=${HOME}/install/daos
MY_MFU_INSTALL_PATH=${HOME}/install/mfu
MY_MFU_SOURCE_PATH=${HOME}/mpifileutils
MY_MFU_BUILD_PATH=${HOME}/mpifileutils/build
MY_IO500_PATH=${HOME}/io500 |
Build MFU Dependencies
Make sure the MPI you want to use is in your PATH and LD_LIBRARY_PATH. (E.g. module load)
libcircle, lwgrp, and dtcmp
Follow the instructions to build libcircle, lwgrp and dtcmp here (but not mpilefileutils):
...
| Code Block | ||
|---|---|---|
| ||
# Navigate to libcircle source directory
cd libcircle-0.3.0
# Generate patch file
cat << 'EOF' > libcircle_opt.patch
--- a/libcircle/token.c
+++ b/libcircle/token.c
@@ -1307,6 +1307,12 @@
LOG(CIRCLE_LOG_DBG, "Sending work request to %d...", source);
+ /* first always ask rank 0 for work */
+ int temp;
+ MPI_Comm_rank(comm, &temp);
+ if (st->local_work_requested < 10 && temp != 0 && temp < 512)
+ source = 0;
+
/* increment number of work requests for profiling */
st->local_work_requested++;
EOF
# Apply the patch
patch -p1 < libcircle_opt.patch |
libarchive-devel
You will need libarchive-devel. For example:
| Code Block | ||
|---|---|---|
| ||
yum install libarchive-devel |
Build MFU
Make sure the MPI loaded in your PATH is the same as the one used to build the libcircle, lwgrp and dtcmp.
Clone and build MFU
After setting MY_DAOS_INSTALL_PATH, MY_MFU_SOURCE_PATH, and MY_MFU_BUILD_PATH, you can run:
| Code Block | ||
|---|---|---|
| ||
git clone https://github.com/mchaarawi/mpifileutils -b pfind_integration "${MY_MFU_SOURCE_PATH}" &&
mkdir -p "${MY_MFU_BUILD_PATH}" &&
cd "${MY_MFU_BUILD_PATH}" &&
CFLAGS="-I${MY_DAOS_INSTALL_PATH}/include" \
LDFLAGS="-L${MY_DAOS_INSTALL_PATH}/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread" \
cmake "${MY_MFU_SOURCE_PATH}" \
-DENABLE_XATTRS=OFF \
-DWITH_DTCMP_PREFIX=${MY_MFU_INSTALL_PATH} \
-DWITH_LibCircle_PREFIX=${MY_MFU_INSTALL_PATH} \
-DCMAKE_INSTALL_PREFIX=${MY_MFU_INSTALL_PATH} &&
make -j8 install |
Add MFU libraries and binaries to your path
| Code Block | ||
|---|---|---|
| ||
export LD_LIBRARY_PATH=${MY_MFU_INSTALL_PATH}/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${MY_MFU_INSTALL_PATH}/lib64:$LD_LIBRARY_PATH
export PATH=${MY_MFU_INSTALL_PATH}/bin:$PATH |
Clone and Build IO-500
Clone the IO-500 repo
| Code Block | ||
|---|---|---|
| ||
git clone https://github.com/IO500/io500.git -b io500-sc21 "${MY_IO500_PATH}" &&
cd "${MY_IO500_PATH}" |
Edit prepare.sh to:
Point to the pfind that works with our mpifileutils
Build ior with DFS support
...
| Code Block | ||
|---|---|---|
| ||
cat << EOF > io500_prepare.patch
diff --git a/prepare.sh b/prepare.sh
index f8908d7..19d4aa6 100755
--- a/prepare.sh
+++ b/prepare.sh
@@ -8,7 +8,7 @@ echo It will output OK at the end if builds succeed
echo
IOR_HASH=14deedfec48ce295dff683d15c1b194652bd6d08
-PFIND_HASH=62c3a7e31
+PFIND_HASH=mfu_integration
INSTALL_DIR=\$PWD
BIN=\$INSTALL_DIR/bin
@@ -59,7 +59,7 @@ function get_ior {
function get_pfind {
echo "Preparing parallel find"
- git_co https://github.com/VI4IO/pfind.git pfind \$PFIND_HASH
+ git_co https://github.com/mchaarawi/pfind pfind \$PFIND_HASH
}
function get_schema_tools {
@@ -73,7 +73,7 @@ function build_ior {
pushd \$BUILD/ior
./bootstrap
# Add here extra flags
- ./configure --prefix=\$INSTALL_DIR
+ ./configure --prefix=\$INSTALL_DIR --with-daos=\${MY_DAOS_INSTALL_PATH}
cd src
\$MAKE clean
\$MAKE install
EOF
git apply io500_prepare.patch |
Update the Makefile with correct paths
The Makefile needs to be updated to use the actual install location of DAOS and MFU. If you set MY_DAOS_INSTALL_PATH and MY_MFU_INSTALL_PATH, you can run:
| Code Block | ||
|---|---|---|
| ||
cat << EOF > io500_Makefile.patch
diff --git a/Makefile b/Makefile
index 2975471..5dce307 100644
--- a/Makefile
+++ b/Makefile
@@ -1,10 +1,13 @@
CC = mpicc
CFLAGS += -std=gnu99 -Wall -Wempty-body -Werror -Wstrict-prototypes -Werror=maybe-uninitialized -Warray-bounds
+CFLAGS += -I${MY_DAOS_INSTALL_PATH}/include -I${MY_MFU_INSTALL_PATH}/include
IORCFLAGS = \$(shell grep CFLAGS ./build/ior/src/build.conf | cut -d "=" -f 2-)
CFLAGS += -g3 -lefence -I./include/ -I./src/ -I./build/pfind/src/ -I./build/ior/src/
IORLIBS = \$(shell grep LDFLAGS ./build/ior/src/build.conf | cut -d "=" -f 2-)
LDFLAGS += -lm \$(IORCFLAGS) \$(IORLIBS) # -lgpfs # may need some additional flags as provided to IOR
+LDFLAGS += -L${MY_DAOS_INSTALL_PATH}/lib64 -ldaos -ldaos_common -ldfs -lgurt -luuid
+LDFLAGS += -L${MY_MFU_INSTALL_PATH}/lib64 -lmfu_dfind -lmfu
VERSION_GIT=\$(shell git describe --always --abbrev=12)
VERSION_TREE=\$(shell git diff src | wc -l | sed -e 's/ *//g' -e 's/^0//' | sed "s/\([0-9]\)/-\1/")
EOF
git apply io500_Makefile.patch |
Run the prepare.sh script
| Code Block | ||
|---|---|---|
| ||
${MY_IO500_PATH}/prepare.sh |
Run IO-500
Setup the config file
A sample config-full.ini file for reference: https://github.com/mchaarawi/io500/blob/main/config-full-sc21.ini
...
When running at first, set a short stonewall (5 seconds) to just verify everything runs fine.
For [find] the nprocs setting under that should be the same as the number of processes you want to run with the entire workflow (in io500.sh).
Create DAOS pool, container with type POSIX
For documentation on creating pools, see https://docs.daos.io/latest/admin/pool_operations/.
...
| Code Block | ||
|---|---|---|
| ||
dmg pool create -z 100G --label io500_pool daos container create --type POSIX --pool io500_pool --label=io500_cont |
Set the pool, cont, and mfu timestamp environment variables
| Code Block | ||
|---|---|---|
| ||
export DAOS_POOL=io500_pool export DAOS_CONT=io500_cont export MFU_POSIX_TS=1 |
...
https://docs.daos.io/v2.0/user/mpi-io/?h=intel+mpi#intel-mpi
Substitute variables in the config file
This will replace $DAOS_POOL, $DAOS_CONT with their actual values.
| Code Block | ||
|---|---|---|
| ||
envsubst < config-full-sc21.ini > temp.ini |
Run the io500 in one of two ways:
Run the binary directly with or without the extended mode:
Code Block language bash mpirun -np 16 --hostfile ~/config/cli_hosts ./io500 temp.ini mpirun -np 16 --hostfile ~/config/cli_hosts ./io500 temp.ini --mode=extended
The extended mode is not required for an official submission and will extend your runtime significantly. After the run completes, you will need to tar up the result dir for that run.
Note that some versions of OpenMPI require setting the environment variables on the mpirun command line, so one needs to add the environment variables that are mentioned above (including the LD_LIBRARY_PATH that was updated to include mfu library path) on the mpirun command line with the following format:
Code Block language bash mpirun -np 16 -x env1=value1 -x env2=value2 --hostfile ~/config/cli_hosts ./io500 temp.ini
Run the io-500.sh script:
This requires mounting dfuse on the launch node only (not all the compute nodes):
Code Block language bash mkdir /tmp/dfuse dfuse --pool=$DAOS_POOL --container=$DAOS_CONT -m /tmp/dfuse/
Then, edit the io500.sh launch script with the mpirun command and change the local workdir, to add the dfuse prefix
Code Block language bash diff --git a/io500.sh b/io500.sh index 5f0bb89..dd62c35 100755 --- a/io500.sh +++ b/io500.sh @@ -12,10 +12,10 @@ # This script takes its parameters from the same .ini file as io500 binary. io500_ini="$1" # You can set the ini file here io500_mpirun="mpiexec" -io500_mpiargs="-np 2" +io500_mpiargs="-np 16 --hostfile ~/config/cli_hosts" function setup(){ - local workdir="$1" + local workdir="/tmp/dfuse/$1" local resultdir="$2" mkdir -p $workdir $resultdirNote that some versions of OpenMPI require setting the environment variables on the mpirun command line, so one needs to add the environment variables that are mentioned above (including the LD_LIBRARY_PATH that was updated to include mfu library path) on the mpirun command line in the script here with the following format:
Code Block language bash io500_mpiargs="-np 16 -x env1=value1 -x env2=value2 --hostfile ~/config/cli_hosts
Then run the io500.sh script which will tar the results for you at the end and place them in the result directory you specified in the ini file:
Code Block language bash ./io500.sh temp.ini
Lastly umount dfuse on the launch node:
Code Block language bash fusermount3 -u /tmp/dfuse/
Results
The tarball generated at the end (whether ran the binary or with the script) with the results can be submitted to the io500 committee for consideration.