...
An attempt to support tiering from the client side was attempted several years ago and exposed a lot of complexity difficult to overcome. For this new approach, it was thus decided to support tiering as a first class feature in DAOS and drive it from the engine which has a finer-grain grained visibility and control on of the data.
...
New daos_mover Service
...
For phase 1, the daos_mover is mostly responsible from for handling cache miss from the engine, converting those request requests into libdfs one requests and copying data from the backend tier to the container. It will act over libdfs as a regular client and need to be able to open the container and access it. It is also acknowledge acknowledged that the daos_mover process/threads on one node can issue write or read request requests to non-local engine engines via libdfs/libdaos. This might be optimised in the future.
...
Upon a cache miss, the data mover will be in charge of pulling the missing data from the backend tier and this might take some time. As a consequence, we cannot afford to just hold the reply until the data is copied since the original RPC might time out on the client. Currently, DAOS has no mechanism to issue notifications/RPCs from a server to a client. This could be implemented by using RDMA and having the server issuing a PUT when the requested data is available and libdaos can retry the request. This mechanism can be used in other featurefeatures, like flock implementation or new snapshot notifications.
...
The data_mover service needs to be provide specific permission over libdfs/daos to be able to access the container, but also to write to objects supposed to be immutable (what a regular client is actually not allowed to do). Rebuild already uses a specific handle that can hopefully be shared (or at least the mechanism) with the data_mover service without messing up with ACLs.
Mapping between DAOS objects and backend objects/files
...