...
Lei Huang, Mohamad Chaarawi, Ashley Pittman
Introduction
...
Applications normally need to be modified by adopting the DFS API to fully take advantage of the performance provided by DAOS. Such requirements need extra effort from software developers for porting and maintaining. For applications that do not support DFS, dfuse is used to support applications using POSIX interface without the need of code changes with degraded I/O performance compared using native DFS API. Interception library libioil.so was introduced to boost read/write bandwidth by bypassing dfuse for read()/write() and their variants. Metadata related functions (e.g., open, stat, unlink, etc.) still suffer from inferior performance restricted by dfuse. A new interception library that intercepts all I/O related functions in glibc (including read/write as well as metadata related functions) is required to deliver good performance using POSIX comparable with using DFS.
Design Overview
The design of this interception library is quite straightforward. We just need a way to implement function interception and provide the new functions for all interested functions in libc, libpthread and ld. To bypass dfuse, we do need to use fake file descriptors (fd) managed in user space. Such fake fd can be assigned as large integers, so they can be distinguished easily from the real fd allocated by Linux kernel. All I/O related functions have to be intercepted since kernel does not recognize the fake fd allocated by our interception library.
...
In principal, we do need to implement all I/O related functions in our interception library since fake fd is used. This means a large number of APIs are involved and a lot of tests will be needed. It may take significant time to be stable and support most applications.
User Interface
In current implementations, simple string comparison for path and integer comparison for fd are used to determine whether file/directory is on DAOS filesystem or not. A list of DAOS container mount points are maintained.
...
export LD_PRELOAD=/usr/lib64/libpil4dfs.so
mpirun -np xx mdtest …
Limitations
Stability issues: Many APIs are involved. We have not carefully tested each function. There may be bugs, uncovered/not intercepted functions, etc.
...