Pre-requisites

DAOS - See https://daos-stack.github.io/admin/hardware/ for installation and setup instructions
MPI - any version / implementation with the devel libraries
cmake 3.1+
clush - See https://clustershell.readthedocs.io/en/latest/install.html for installation
- Alternatives are possible, though examples are not provided in these instructions.

Build Paths

These instructions assume the following paths. For simplicity, you can set these variables to the actual locations where you have/want these installed.

After setting these variables, most of the scripts can be "copy-pasted".

MY_DAOS_INSTALL_PATH=${HOME}/install/daos
MY_MFU_INSTALL_PATH=${HOME}/install/mfu
MY_MFU_SOURCE_PATH=${HOME}/mpifileutils
MY_MFU_BUILD_PATH=${HOME}/mpifileutils/build
MY_IO500_PATH=${HOME}/io500

Build MFU Dependencies

Make sure the MPI you want to use is in your PATH and LD_LIBRARY_PATH. (E.g. module load)

libcircle, lwgrp, and dtcmp

Follow the instructions to build libcircle, lwgrp and dtcmp here (but not mpilefileutils):

https://mpifileutils.readthedocs.io/en/v0.10.1/build.html#build-everything-directly

You can use the same prefix for all three dependencies above (libcircle, lwgrp, dtcmp). Let’s assume we used: ${MY_MFU_INSTALL_PATH}

Before building libcircle, an optimization you can make is to apply this change to the source (the 512 in there can be tuned more depending on how many total MPI ranks you are using to run the io-500):

# Navigate to libcircle source directory
cd libcircle-0.3.0

# Generate patch file
cat << 'EOF' > libcircle_opt.patch
--- a/libcircle/token.c
+++ b/libcircle/token.c
@@ -1307,6 +1307,12 @@

         LOG(CIRCLE_LOG_DBG, "Sending work request to %d...", source);

+        /* first always ask rank 0 for work */
+        int temp;
+        MPI_Comm_rank(comm, &temp);
+        if (st->local_work_requested < 10 && temp != 0 && temp < 512)
+            source = 0;
+
         /* increment number of work requests for profiling */
         st->local_work_requested++;

EOF

# Apply the patch
patch -p1 < libcircle_opt.patch

libarchive-devel

You will need libarchive-devel. For example:

yum install libarchive-devel

Build MFU

Make sure the MPI loaded in your PATH is the same as the one used to build the libcircle, lwgrp and dtcmp.

Clone and build MFU

Aftet setting MY_DAOS_INSTALL_PATH, MY_MFU_SOURCE_PATH, and MY_MFU_BUILD_PATH, you can run:

git clone https://github.com/mchaarawi/mpifileutils -b pfind_integration "${MY_MFU_SOURCE_PATH}" &&
mkdir -p "${MY_MFU_BUILD_PATH}" &&
cd "${MY_MFU_BUILD_PATH}" &&
CFLAGS="-I${MY_DAOS_INSTALL_PATH}/include" \
LDFLAGS="-L${MY_DAOS_INSTALL_PATH}/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread" \
cmake "${MY_MFU_SOURCE_PATH}" \
  -DENABLE_XATTRS=OFF \
  -DWITH_DTCMP_PREFIX=${MY_MFU_INSTALL_PATH} \
  -DWITH_LibCircle_PREFIX=${MY_MFU_INSTALL_PATH} \
  -DCMAKE_INSTALL_PREFIX=${MY_MFU_INSTALL_PATH} &&
make -j8 install

Add MFU libraries and binaries to your path

export LD_LIBRARY_PATH=${MY_MFU_INSTALL_PATH}/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${MY_MFU_INSTALL_PATH}/lib64:$LD_LIBRARY_PATH
export PATH=${MY_MFU_INSTALL_PATH}/bin:$PATH

Clone and Build IO-500

Clone the IO-500 repo

git clone https://github.com/IO500/io500.git -b io500-isc21 "${MY_IO500_PATH}" &&
cd "${MY_IO500_PATH}"

Edit prepare.sh to:

Point to the pfind that works with our mpifileutils
Build ior with DFS support

Assuming MY_DAOS_INSTALL_PATH is set, you can run:

cat << EOF > io500_prepare.patch
diff --git a/prepare.sh b/prepare.sh
index f8908d7..19d4aa6 100755
--- a/prepare.sh
+++ b/prepare.sh
@@ -8,7 +8,7 @@ echo It will output OK at the end if builds succeed
 echo

 IOR_HASH=0410a38e985e0862a9fd9abec017abffc4c5fc43
-PFIND_HASH=62c3a7e31
+PFIND_HASH=mfu_integration

 INSTALL_DIR=\$PWD
 BIN=\$INSTALL_DIR/bin
@@ -59,7 +59,7 @@ function get_ior {

 function get_pfind {
   echo "Preparing parallel find"
-  git_co https://github.com/VI4IO/pfind.git pfind \$PFIND_HASH
+  git_co https://github.com/mchaarawi/pfind pfind \$PFIND_HASH
 }

 function get_schema_tools {
@@ -73,7 +73,7 @@ function build_ior {
   pushd \$BUILD/ior
   ./bootstrap
   # Add here extra flags
-  ./configure --prefix=\$INSTALL_DIR
+  ./configure --prefix=\$INSTALL_DIR --with-daos=${MY_DAOS_INSTALL_PATH}
   cd src
   \$MAKE clean
   \$MAKE install
EOF

git apply io500_prepare.patch

Update the Makefile with correct paths

The Makefile needs to be updated to use the actual install location of DAOS and MFU. If you set MY_DAOS_INSTALL_PATH and MY_MFU_INSTALL_PATH, you can run:

cat << EOF > io500_Makefile.patch
diff --git a/Makefile b/Makefile
index 2975471..5dce307 100644
--- a/Makefile
+++ b/Makefile
@@ -1,10 +1,13 @@
 CC = mpicc
 CFLAGS += -std=gnu99 -Wall -Wempty-body -Werror -Wstrict-prototypes -Werror=maybe-uninitialized -Warray-bounds
+CFLAGS += -I${MY_DAOS_INSTALL_PATH}/include -I${MY_MFU_INSTALL_PATH}/include

 IORCFLAGS = \$(shell grep CFLAGS ./build/ior/src/build.conf | cut -d "=" -f 2-)
 CFLAGS += -g3 -lefence -I./include/ -I./src/ -I./build/pfind/src/ -I./build/ior/src/
 IORLIBS = \$(shell grep LDFLAGS ./build/ior/src/build.conf | cut -d "=" -f 2-)
 LDFLAGS += -lm \$(IORCFLAGS) \$(IORLIBS) # -lgpfs # may need some additional flags as provided to IOR
+LDFLAGS += -L${MY_DAOS_INSTALL_PATH}/lib64 -ldaos -ldaos_common -ldfs -lgurt -luuid
+LDFLAGS += -L${MY_MFU_INSTALL_PATH}/lib64 -lmfu_dfind -lmfu

 VERSION_GIT=\$(shell git describe --always --abbrev=12)
 VERSION_TREE=\$(shell git diff src | wc -l | sed -e 's/   *//g' -e 's/^0//' | sed "s/\([0-9]\)/-\1/")
EOF

git apply io500_Makefile.patch

Run the prepare.sh script

${MY_IO500_PATH}/prepare.sh

Run io-500

Setup the config file

A sample config-full.ini file for reference: https://github.com/mchaarawi/io500/blob/main/config-full-isc21.ini

If you want to download this:

wget https://raw.githubusercontent.com/mchaarawi/io500/main/config-full-isc21.ini

When running at first, set a short stonewall (5 seconds) to just verify everything runs fine.

For [find] the nprocs setting under that should be the same as the number of processes you want to run with the entire workflow (in io500.sh).

Create DAOS pool, container with type POSIX

The pool size depends on the settings you use in the config file - larger datasets require more pool space.

For documentation on creating pools, see https://daos-stack.github.io/admin/pool_operations/.

For documentation on creating containers, see https://daos-stack.github.io/user/container/.

For example:

dmg pool create -z 100G --label io500_pool
daos container create --type POSIX --pool io500_pool

Set the pool, cont, fuse environment variables

export DAOS_POOL=<uuid>
export DAOS_CONT=<uuid> 
export DAOS_FUSE=<path>

Create a dfuse mount on all the compute nodes

clush --hostfile cli_hosts "dfuse --pool=$DAOS_POOL --container=$DAOS_CONT -m $DAOS_FUSE"

Substitute variables in the config file

This will replace $DAOS_POOL, $DAOS_CONT, and $DAOS_FUSE with their actual values.

envsubst < config-full-isc21.ini > temp.ini

Update io500.sh script with the mpirun command and number of processes to use and run it:

./io500.sh temp.ini

Results

A tarball is generated at the end with the results that you can submit to the io500 committee for consideration.

DAOS Community

IO-500 ISC21

Table of Contents