Replace a rank when metadata is lost

(A problem exists whereby the number of ranks increments every time the metadata stored on a rank’s SCM is lost. The cause of the problem is that the rank-UUID mapping in the MS is fixed and so when the old UUID is lost we can no longer map the engine to that rank.

The problem scenario is illustrated as follows:

1. A PMem module fails and metadata stored on SCM is lost, the rank gets excluded and the targets distributed to other ranks. From then on that rank will always be marked as excluded in the system query output because the UUID mapped to it is no longer known to the engine.

2. The engine’s storage needs to be reformatted once the faulty PMem module has been replaced and both AppDirect region and /dev/pmem namespace recreated. The new engine then joins as a different rank (mapped to a new UUID).

3. We then use pool extend to add the new rank to the pool as the old rank cannot be used.

The main disadvantages related to the increase in the number of pool ranks over time are:

• The system query dmg output becomes unusable because it gets hard to discern new temporal excluded ranks from those that have been “lost” due to hardware failures.

• Pool-map growth results in metadata size bloat which will eventually make the system sluggish.

• Pool-queries will report disabled targets related to “lost” ranks making it hard for administrators to tell if pools are healthy or degraded.

The solution to the aforementioned problem is to add the capability to replace a rank by updating the UUID used to map it rather than having to create a new one. This enables the same engine to rejoin a system with it’s previous rank identifier after a PMem replacement.

The new process for replacing a rank in this case requires the dmg storage format --replace flag to be passed during SCM format. The following example emulates the erasure of storage that would occur with recreation of AppDirect interleaved mode and PMem namespace after bad NVDIMM replacement.

1. Use dmg storage scan -v to identify the NUMA/Socket ID of rank to stop.

2. Stop "bad" rank 4.

3. Stop remote daos_server that is running rank 4. (This would be done as part of PMem NVDIMM replacement). Observe rank 3 and 4 are both excluded.

4. Wipe pmem device on remote storage server engine with rank 4. (Emulating PMem NVDIMM replacement)

5. Start remote daos_server that was running rank 4. Observe rank 3 joins successfully.

6. Run dmg storage format --replace -l wolf-312 to re-join rank 4. (Note single hostname specifier compulsory when using the replace flag). Observe rank 4 joins successfully (and no new rank entry is added).

As a note, the control log entries for processing of "replace" join request contain a pool-query for each pool relevant to the replaced rank in order to verify it’s excluded from every pool before updating its UUID in the system membership.

7. Reintegrate ranks back into their previous pools. (Note that dmg system reint --ranks 3-4 can be replaced by dmg system reint --host-ranks wolf-312 to select the ranks resident on a given storage host)

Notes:

• Restriction exists in control-plane format logic that prevents a rank which resides on any MS replica (list specified in server config file ms_replicas parameter) host/storage-server from being replaced in this manner. Storage format command will return an error stating that “storage format invoked on a running system”. Reference: https://daosio.atlassian.net/browse/DAOS-17385

• In certain situations both ranks on a dual-engine server need to be replaced, if they are both in the correct state (i.e. PMem wiped and engine spec matches an entry in the MS-db) then both engines will rejoin with pre-existing ranks and output will look something like:

  [daos]$ install/bin/dmg storage format --replace -l wolf-312 -v -------- wolf-312 -------- SCM Mount Format Result --------- ------------- /mnt/daos0 CTL_SUCCESS /mnt/daos1 CTL_SUCCESS ...