IO-500 with the POSIX API

Owned by Mohamad Chaarawi
Last updated: Jul 08, 2024 by Michael Hennecke
3 min read

Running the IO-500 with the POSIX API and getting similar performance to when running with the -a DFS API, requires a few extra / different steps:

  1. Build the IO-500 the same way as indicated in the parent page here.

    • Note that updating the prepare and makefile is not really required but preferred, just to provide the option of running using either APIs (POSIX and DFS) to compare performance while the same io500 binary.

  2. Create the pool the same way, but change the default oclass on the container:

dmg pool create -z=100% io500_pool daos container create --type POSIX --file-oclass=S1 --dir-oclass=SX io500_pool io500_cont

  • When creating the container with different redundancy factors than RF:0 (e.g. by specifying --properties rd_fac:2 with daos cont create), change the file-oclass to RPnG1 and dir-oclass to RPnGX; where n depends on the redundancy factor you need: (RF:1, n=2 ; RF:2, n=3);

  1. Mount the container with dfuse on all the client nodes. You can use clush or pdsh, etc. In this example we mount the container on /tmp/dfuse (does not have to be that location):

clush --hostfile ~/path/to/cli_hosts "mkdir -p /tmp/dfuse; dfuse --disable-caching /tmp/dfuse/ io500_pool io500_cont"

  1. Update the io500.sh script:

    1. Change the MPI run command to add LD_PRELOAD for using the interception library and the DAOS env variables for the pfind app (make sure to update the pool, container, and dfuse mount for the DAOS_PREFIX, as well as the path for the pil4dfs interception library) (Note: the options to the mpirun command are specific to mpich, and other MPI implementation have different ways to pass environment variables; like -x for open-mpi for example):

    2. io500_mpirun="mpirun -genv LD_PRELOAD=/scratchbox/daos/mschaara/install/daos/lib64/libpil4dfs.so -genv DAOS_POOL=io500_pool -genv DAOS_CONT=io500_cont -genv DAOS_PREFIX=/tmp/dfuse"

    3. Pre-create the ior-easy and ior-hard directories in the setup() function to change the oclass of those to be different than the container default (need to be widely striped for better BW). Change the oclass below to an EC oclass with GX if you are using rf > 0:

  2. Use an io500 ini file with the POSIX API. An example is provided below.

    1. Update the nproc for find to be the same number of procs you used in the the io500.sh script.

    2. Update resultdir to where you want to store the result tarball.

    3. For ior-easy you can always change the file-per-proc and transfer size to something that suits better your configuration.