IO500 Run

Owned by Dalton Bohning
Last updated: Oct 22, 2021

Table of Contents

Introduction

The I500 benchmark measures the performance of various workloads, including ior, mdtest, and find. For general information about IO500, see IO500 - About.

This document provides instructions to build the components necessary to run the IO500 benchmark with DAOS.

Requirements

Set environment variables for list of servers, client and admin node.
export ADMIN_NODE=node-1
export SERVER_NODES=node-2,node-3
export CLIENT_NODES=node-4,node-5
export ALL_NODES=$ADMIN_NODE,$SERVER_NODES,$CLIENT_NODES
Install pdsh on admin node
sudo yum install -y pdsh
MPI

Any implementation or version of MPI should work, as long as the same MPI is used for all instructions and the MPI devel packages are installed.. For example:

# load openmpi module or set its path in your environment
module load mpi/openmpi3-x86_64
or
export LD_LIBRARY_PATH=<openmpi lib path>:$LD_LIBRARY_PATH
export PATH=<openmpi bin path>:$PATH
Install dependent packages on client nodes

For Centos:

sudo yum install -y cmake3 libarchive-devel patch automake autoconf gcc-c++ libuuid-devel bzip2-devel openssl-devel Lmod

For openSUSE/SLES 15.2:

sudo zypper install -y cmake3 libarchive-devel patch automake autoconf gcc-c++ libuuid-devel libbz2-devel libopenssl-devel lua-lmod

Set-Up

Please refer to DAOS Set-Up for initial set up which consists of rpm installation, generate and set up certificates, setting up config files, starting servers and agents.

For setting up on a specific system, a Quick Start guide can be followed:

If installing from packages/RPMS daos-devel is also required on all $CLIENT_NODES:

sudo yum install -y daos-devel

Environment Paths

These environment variables should be set to the actual locations where you have/want each installed.

After setting these variables, most of the scripts can be "copy-pasted".

# Where DAOS is installed. Use /usr if installed from RPMs.
MY_DAOS_INSTALL_PATH=/usr

# Where the mpifileutils repository will be cloned to
MY_MFU_SOURCE_PATH=${HOME}/mpifileutils

# Where to build dependencies for mpifileutils
MY_MFU_DEPS_PATH=${MY_MFU_SOURCE_PATH}/deps

# Where to build mpifileutils
MY_MFU_BUILD_PATH=${MY_MFU_SOURCE_PATH}/build

# Where to install the mpifileutils dependencies, libraries, and binaries
MY_MFU_INSTALL_PATH=${MY_MFU_SOURCE_PATH}/install

# Where the io500 repository will be cloned to
MY_IO500_PATH=${HOME}/io500

Build MpiFileUtils and Dependencies

These dependencies should be installed on all $CLIENT_NODES.

Clone MpiFileUtils repository
git clone https://github.com/mchaarawi/mpifileutils -b pfind_integration "${MY_MFU_SOURCE_PATH}"
Create build and install directories
mkdir -p ${MY_MFU_DEPS_PATH} ${MY_MFU_BUILD_PATH} ${MY_MFU_INSTALL_PATH}
Build MpiFileUtils dependencies
cd ${MY_MFU_DEPS_PATH}
  wget https://github.com/hpc/libcircle/releases/download/v0.3/libcircle-0.3.0.tar.gz
  wget https://github.com/llnl/lwgrp/releases/download/v1.0.3/lwgrp-1.0.3.tar.gz
  wget https://github.com/llnl/dtcmp/releases/download/v1.1.1/dtcmp-1.1.1.tar.gz

  tar -zxf libcircle-0.3.0.tar.gz
  cd libcircle-0.3.0
# OPTIONAL - This optional optimization can be applied to libcircle before building.
# The "512" can be tuned depending on how many total MPI ranks are used to run the io-500.
cat << 'EOF' > libcircle_opt.patch
--- a/libcircle/token.c
+++ b/libcircle/token.c
@@ -1307,6 +1307,12 @@

         LOG(CIRCLE_LOG_DBG, "Sending work request to %d...", source);

+        /* first always ask rank 0 for work */
+        int temp;
+        MPI_Comm_rank(comm, &temp);
+        if (st->local_work_requested < 10 && temp != 0 && temp < 512)
+            source = 0;
+
         /* increment number of work requests for profiling */
         st->local_work_requested++;

EOF
patch -p1 < libcircle_opt.patch
# END optional patch
    ./configure --prefix=${MY_MFU_INSTALL_PATH}
    make install
  cd ..

  tar -zxf lwgrp-1.0.3.tar.gz
  cd lwgrp-1.0.3
    ./configure --prefix=${MY_MFU_INSTALL_PATH}
    make install
  cd ..

  tar -zxf dtcmp-1.1.1.tar.gz
  cd dtcmp-1.1.1
    ./configure --prefix=${MY_MFU_INSTALL_PATH} --with-lwgrp=${MY_MFU_INSTALL_PATH}
    make install
  cd ..
cd ..
Build and install MpiFileUtils

When building mpifileutils, the CFLAGS and LDFLAGS need to be set on the same line as the cmake3 command for the variables to be propagated to the cmake environment.

cd "${MY_MFU_BUILD_PATH}" &&
CFLAGS="-I${MY_DAOS_INSTALL_PATH}/include" \
LDFLAGS="-L${MY_DAOS_INSTALL_PATH}/lib64/ -luuid -ldaos -ldfs -ldaos_common -lgurt -lpthread" \
cmake3 "${MY_MFU_SOURCE_PATH}" \
  -DENABLE_XATTRS=OFF \
  -DWITH_DTCMP_PREFIX=${MY_MFU_INSTALL_PATH} \
  -DWITH_LibCircle_PREFIX=${MY_MFU_INSTALL_PATH} \
  -DCMAKE_INSTALL_PREFIX=${MY_MFU_INSTALL_PATH} &&
make -j8 install
Add MpiFileUtils libraries and binaries to your path
export LD_LIBRARY_PATH=${MY_MFU_INSTALL_PATH}/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${MY_MFU_INSTALL_PATH}/lib64:$LD_LIBRARY_PATH
export PATH=${MY_MFU_INSTALL_PATH}/bin:$PATH

Clone and Build IO-500

Clone the IO-500 repo

git clone https://github.com/IO500/io500.git -b io500-isc21 "${MY_IO500_PATH}" &&
cd "${MY_IO500_PATH}"

Edit prepare.sh to:

  • Point to the pfind that works with our mpifileutils
  • Use the latest master of IOR (you should use whatever the io-500 committee has for that submission though)
  • Build ior with DFS support

Assuming MY_DAOS_INSTALL_PATH is set, you can run:

cat << EOF > io500_prepare.patch
diff --git a/prepare.sh b/prepare.sh
index f8908d7..19d4aa6 100755
--- a/prepare.sh
+++ b/prepare.sh
@@ -8,7 +8,7 @@ echo It will output OK at the end if builds succeed
 echo

 IOR_HASH=0410a38e985e0862a9fd9abec017abffc4c5fc43
-PFIND_HASH=62c3a7e31
+PFIND_HASH=mfu_integration

 INSTALL_DIR=\$PWD
 BIN=\$INSTALL_DIR/bin
@@ -59,7 +59,7 @@ function get_ior {

 function get_pfind {
   echo "Preparing parallel find"
-  git_co https://github.com/VI4IO/pfind.git pfind \$PFIND_HASH
+  git_co https://github.com/mchaarawi/pfind pfind \$PFIND_HASH
 }

 function get_schema_tools {
@@ -73,7 +73,7 @@ function build_ior {
   pushd \$BUILD/ior
   ./bootstrap
   # Add here extra flags
-  ./configure --prefix=\$INSTALL_DIR
+  ./configure --prefix=\$INSTALL_DIR --with-daos=${MY_DAOS_INSTALL_PATH}
   cd src
   \$MAKE clean
   \$MAKE install
EOF

git apply io500_prepare.patch

Update the Makefile with correct paths

The Makefile needs to be updated to use the actual install location of DAOS and MFU. If you set MY_DAOS_INSTALL_PATH and MY_MFU_INSTALL_PATH, you can run:

cat << EOF > io500_Makefile.patch
diff --git a/Makefile b/Makefile
index 2975471..5dce307 100644
--- a/Makefile
+++ b/Makefile
@@ -1,10 +1,13 @@
 CC = mpicc
 CFLAGS += -std=gnu99 -Wall -Wempty-body -Werror -Wstrict-prototypes -Werror=maybe-uninitialized -Warray-bounds
+CFLAGS += -I${MY_DAOS_INSTALL_PATH}/include -I${MY_MFU_INSTALL_PATH}/include

 IORCFLAGS = \$(shell grep CFLAGS ./build/ior/src/build.conf | cut -d "=" -f 2-)
 CFLAGS += -g3 -lefence -I./include/ -I./src/ -I./build/pfind/src/ -I./build/ior/src/
 IORLIBS = \$(shell grep LDFLAGS ./build/ior/src/build.conf | cut -d "=" -f 2-)
 LDFLAGS += -lm \$(IORCFLAGS) \$(IORLIBS) # -lgpfs # may need some additional flags as provided to IOR
+LDFLAGS += -L${MY_DAOS_INSTALL_PATH}/lib64 -ldaos -ldaos_common -ldfs -lgurt -luuid
+LDFLAGS += -L${MY_MFU_INSTALL_PATH}/lib64 -lmfu_dfind -lmfu

 VERSION_GIT=\$(shell git describe --always --abbrev=12)
 VERSION_TREE=\$(shell git diff src | wc -l | sed -e 's/   *//g' -e 's/^0//' | sed "s/\([0-9]\)/-\1/")
EOF

git apply io500_Makefile.patch

Run the prepare.sh script

${MY_IO500_PATH}/prepare.sh

Edit compile.sh

It is expected here that IOR builds with the DFS driver, but pfind build will fail.

build/pfind/compile.sh needs to be updated to point to the correct DAOS and MFU paths.

DAOS=${MY_DAOS_INSTALL_PATH}
MFU=${MY_MFU_INSTALL_PATH}

To update, you can run:

sed -i "/^DAOS=/c\DAOS=${MY_DAOS_INSTALL_PATH}" ${MY_IO500_PATH}/build/pfind/compile.sh
sed -i "/^MFU=/c\MFU=${MY_MFU_INSTALL_PATH}" ${MY_IO500_PATH}/build/pfind/compile.sh

Then run prepare.sh again:

${MY_IO500_PATH}/prepare.sh

Run io-500

Setup the config file

A sample config-full.ini file for reference: https://github.com/mchaarawi/io500/blob/main/config-full.ini

If you want to download this:

wget https://raw.githubusercontent.com/mchaarawi/io500/main/config-full.ini

In DAOS 1.1.3 and above, svcl is not required anymore and should be removed from the config.

sed -i 's/ --dfs.svcl=$DAOS_SVCL//g' config-full.ini

You should set a low stonewall at first to verify.

For [find] the nprocs setting under that should be the same as the number of processes you want to run with the entire workflow.

Create DAOS pool, container with type POSIX

The pool size depends on the settings you use in the config file - larger datasets require more pool space.

For documentation on creating pools, see https://daos-stack.github.io/admin/pool_operations/.

For documentation on creating containers, see https://daos-stack.github.io/user/container/.

For example:

dmg pool create -z 100G --label io500_pool
daos container create --type POSIX --pool io500_pool

Set the pool, cont, fuse environment variables

This should be the pool, container, and dfuse path to use for the IO500 run.

export DAOS_POOL=<uuid>
export DAOS_CONT=<uuid> 
export DAOS_FUSE=<path>

# Only for DAOS version < 1.1.3
DAOS_SVCL=<svcl>

Create a dfuse mount on all client nodes

pdsh -w $CLIENT_NODES -f 4 "mkdir -p $DAOS_FUSE && dfuse --pool=$DAOS_POOL --container=$DAOS_CONT -m $DAOS_FUSE"

Substitute variables in the config file

This assumes you are in the directory ${MY_IO500_PATH}.

This will replace $DAOS_POOL, $DAOS_CONT, and $DAOS_FUSE with their actual values from the environment.

envsubst < config-full.ini > temp.ini

Update io500.sh script  with the mpirun command and number of processes to use

Depending on the MPI used, different flags may be required.

For openmpi:

sed -i "s/io500_mpiargs=".*"/io500_mpiargs=\"-hosts $CLIENT_NODES -x DAOS_POOL -x DAOS_CONT -x LD_LIBRARY_PATH -x PATH -np 2\"/g" io500.sh

For mpich:

sed -i "s/io500_mpiargs=".*"/io500_mpiargs=\"-hosts $CLIENT_NODES -np 2\"/g" io500.sh

Run the io500 script

./io500.sh temp.ini

Results

After running, the results will be in the $DAOS_FUSE/results directory.

  • result_summary.txt - Summary of IO rates and times for each test.
  • result.txt - Each test command that was ran, along with rates and times.

To view comparable results, you can download any of the official DAOS submissions here: https://io500.org/.