IO-500 SC20

Table of Contents


Build Paths

These instructions assume the following paths. For simplicity, you can set these variables to the actual locations where you have/want these installed.

After setting these variables, most of the scripts can be "copy-pasted".


Build MFU Dependencies

Make sure the MPI you want to use is in your PATH and LD_LIBRARY_PATH. (E.g. module load)

libcirlce, lwgrp, and dtcmp

Follow the instructions to build libcircle, lwgrp and dtcmp here (but not daos):

You can use the same prefix for all three dependencies above (libcircle, lwgrp, dtcmp). Let’s assume we used: ${MY_MFU_INSTALL_PATH}

Before building libcircle, an optimization you can make is to apply this change to the source (the 512 in there can be tuned more depending on how many total MPI ranks you are using to run the io-500):

# Navigate to libcircle source directory
cd libcircle-0.3.0

# Generate patch file
cat << 'EOF' > libcircle_opt.patch
--- a/libcircle/token.c
+++ b/libcircle/token.c
@@ -1307,6 +1307,12 @@

         LOG(CIRCLE_LOG_DBG, "Sending work request to %d...", source);

+        /* first always ask rank 0 for work */
+        int temp;
+        MPI_Comm_rank(comm, &temp);
+        if (st->local_work_requested < 10 && temp != 0 && temp < 512)
+            source = 0;
         /* increment number of work requests for profiling */


# Apply the patch
patch -p1 < libcircle_opt.patch


You will need libarchive-devel. For example:

yum install libarchive-devel

Build MFU

Make sure the MPI loaded in your PATH is the same as the one used to build the libcircle, lwgrp and dtcmp.

Clone and build MFU


git clone -b pfind_integration "${MY_MFU_SOURCE_PATH}" &&
mkdir -p "${MY_MFU_BUILD_PATH}" &&
cd "${MY_MFU_BUILD_PATH}" &&
LDFLAGS="-L${MY_DAOS_INSTALL_PATH}/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread" \
cmake "${MY_MFU_SOURCE_PATH}" \
make -j8 install

Add MFU libraries and binaries to your path


Clone and Build IO-500

Clone the IO-500 repo

git clone -b io500-sc20 "${MY_IO500_PATH}" &&
cd "${MY_IO500_PATH}"

Edit to:

  • Point to the pfind that works with our mpifileutils
  • Use the latest master of IOR (you should use whatever the io-500 committee has for that submission though)
  • Build ior with DFS support

Assuming MY_DAOS_INSTALL_PATH is set, you can run:

cat << EOF > io500_prepare.patch
diff --git a/ b/
index de354ee..a2964d7 100755
--- a/
+++ b/
@@ -7,8 +7,8 @@ echo It will also attempt to build the benchmarks
 echo It will output OK at the end if builds succeed


@@ -59,14 +59,14 @@ function get_ior {

 function get_pfind {
   echo "Preparing parallel find"
-  git_co pfind \$PFIND_HASH
+  git_co pfind \$PFIND_HASH

 function build_ior {
   pushd \$BUILD/ior
-  ./configure --prefix=\$INSTALL_DIR
+  ./configure --prefix=\$INSTALL_DIR --with-daos=${MY_DAOS_INSTALL_PATH}
   cd src
   \$MAKE clean
   \$MAKE install

git apply io500_prepare.patch

Update the Makefile with correct paths

The Makefile needs to be updated to use the actual install location of DAOS and MFU. If you set MY_DAOS_INSTALL_PATH and MY_MFU_INSTALL_PATH, you can run:

cat << EOF > io500_Makefile.patch
diff --git a/Makefile b/Makefile
index 2975471..5dce307 100644
--- a/Makefile
+++ b/Makefile
@@ -1,10 +1,13 @@
 CC = mpicc
 CFLAGS += -std=gnu99 -Wall -Wempty-body -Werror -Wstrict-prototypes -Werror=maybe-uninitialized -Warray-bounds

 IORCFLAGS = \$(shell grep CFLAGS ./build/ior/Makefile | cut -d "=" -f 2-)
 CFLAGS += -g3 -lefence -I./include/ -I./src/ -I./build/pfind/src/ -I./build/ior/src/
 IORLIBS = \$(shell grep LIBS ./build/ior/Makefile | cut -d "=" -f 2-)
 LDFLAGS += -lm \$(IORCFLAGS) \$(IORLIBS) # -lgpfs # may need some additional flags as provided to IOR
+LDFLAGS += -L${MY_DAOS_INSTALL_PATH}/lib64 -ldaos -ldaos_common -ldfs -lgurt -luuid
+LDFLAGS += -L${MY_MFU_INSTALL_PATH}/lib64 -lmfu_dfind -lmfu

 VERSION_GIT=\$(shell git describe --always --abbrev=12)
 VERSION_TREE=\$(shell git diff src | wc -l | sed -e 's/   *//g' -e 's/^0//' | sed "s/\([0-9]\)/-\1/")

git apply io500_Makefile.patch

Run the script



It is expected here that IOR builds with the DFS driver, but pfind build will fail.

build/pfind/ needs to be updated to point to the correct DAOS and MFU paths.


To update, you can run:

sed -i "/^DAOS=/c\DAOS=${MY_DAOS_INSTALL_PATH}" ${MY_IO500_PATH}/build/pfind/
sed -i "/^MFU=/c\MFU=${MY_MFU_INSTALL_PATH}" ${MY_IO500_PATH}/build/pfind/

Then run again:


Run io-500

Setup the config file

A sample config-full.ini file for reference:

If you want to download this:


If you are using DAOS 1.1.3 and above, the svcl is not required anymore – need to update that file to remove SVCL env.

This can be done with:

sed -i 's/ --dfs.svcl=$DAOS_SVCL//g' config-full.ini

You should set a low stonewall at first to verify.

For [find] the nprocs setting under that should be the same as the number of processes you want to run with the entire workflow.

Create DAOS pool, container with type POSIX

The pool size depends on the settings you use in the config file - larger datasets require more pool space.

For documentation on creating pools, see

For documentation on creating containers, see

For example:

dmg pool create -z 100G
daos container create --type POSIX --pool $DAOS_POOL

Set the pool, cont, fuse environment variables

export DAOS_POOL=<uuid>
export DAOS_CONT=<uuid> 
export DAOS_FUSE=<path>

# Only for DAOS version < 1.1.3

Create a dfuse mount on all the compute nodes

clush --hostfile ~/config/cli_hosts -f 4 "dfuse --pool=$DAOS_POOL --container=$DAOS_CONT -m $DAOS_FUSE"

For two nodes, an example hostfile might look like:

Alternatively, you can specify the nodes as such:

clush -w "," -f 4 "dfuse --pool=$DAOS_POOL --container=$DAOS_CONT -m $DAOS_FUSE"

Substitute variables in the config file

This will replace $DAOS_POOL, $DAOS_CONT, and $DAOS_FUSE with their actual values.

envsubst < config-full.ini > temp.ini

Either run the app manually:

Using the host file:

mpirun -f ~/config/cli_hosts -np 2 ./io500 temp.ini

Using a host list:

mpirun -hosts "," -np 2 ./io500 temp.ini

OR use their provided script (which generates the tarball):

(note that here you must change the mpi procs / command to whatever you are using)

./ temp.ini


After running, the results will be in the $DAOS_FUSE/results directory.

  • result_summary.txt - Summary of IO rates and times for each test.
  • result.txt - Each test command that was ran, along with rates and times.