MPI-IO DAOS Driver (Setup Guide)

Owned by Mohamad Chaarawi
Last updated: Mar 27, 2020

The code of the MPI-IO ROMIO ADIO driver is in an MPICH fork on github:

https://github.com/daos-stack/mpich

git submodule init

git submodule update


To build on Boro:

  • export MPI_LIB=""
  • might need to update your autotools (autoconf, automake, libtool) as required by mpich.
  • ./autogen.sh
  • mkdir build; cd build

  • ../configure --prefix=dir --enable-fortran=all --enable-romio --enable-cxx --enable-g=all --enable-debuginfo --with-file-system=ufs+daos --with-daos=dir --with-cart=dir

  • add also --with-device=ch3:sock To be able to run with PSM2 provider we need to force mpich to use sockets instead of PSM2

  • make -j8; make install

  • Switch PATH and LD_LIBRARY_PATH where you want to build your client apps or libs that use MPI to the above installed MPICH. 

To disable debugging and get better for performance from mpich, remove those options:

--enable-g=all --enable-debuginfo

and add:

--enable-fast=O3,ndebug --disable-error-checking --without-timing --without-mpit-pvars


Build any client (HDF5, ior, mpi test suites) normally with the mpicc and mpich library we installed above (see child pages).

To run an example:

  1. In one shell Launch DAOS server(s) with orterun (the PATH and LD_LIBRARY_PATH should point to the ompi installation that is installed to the DAOS installation):
    orterun --enable-recovery -np 1 --hostfile ~/my_hosts --report-uri ~/uri.txt /path/to/daos/bin/daos_server -c 8 -a /home/mschaara/
    (the -a path will create a connect file that the clients will use to connect to the DAOS server in singleton mode. so the path should accessible for the client and server.)

  2. At the client side, the following environment variables need to be set:

    1. export PATH=/path/to/mpich/install/bin:$PATH

    2. export LD_LIBRARY_PATH=/path/to/mpich/install/lib:$LD_LIBRARY_PATH

  3. create a DAOS pool with dmg (see DAOS user guide)

  4. export DAOS_POOL=pool_uuid; export DAOS_SVCL=svc_leader(s)
  5. Optionally create a POSIX type DAOS container you want to use and export DAOS_CONT=cont_uuid

    1. daos cont create --pool=$DAOS_POOL --svc=$DAOS_SVCL --type=POSIX

  6. run the client application or test (see child pages for examples). 

Limitations to the current implementation include:

  • Reading Holes does not return 0, but leaves the buffer untouched (Not sure how to fix this - might need to wait for DAOS implementation of iod_map_t to determine holes vs written bytes in the Array extent).
  • No support for MPI file atomicity, preallocate, shared file pointers.