...
The management service regularly issues an iv broadcast across all the engines to push a new lease deadline
If a server passes the lease deadline, it tries to read the iv variable storing the latest deadline from its parent. If the parent does not reply, it tries to read from the iv root.
A node is allowed to serve I/Os until the latest lease deadline + lease time.
Upon exclusion, the node is marked as down in the pool map immediately and rebuild can proceed. That being said, new writes issued from client nodes to objects impacted by the rebuild should be delayed until the lease time has passed.
The extra iv_fetch() from the parent and extra lease time is expected to cover the use case where the parent in the IV tree died and did not propagate the new lease time to its children. It might be avoided altogether if the IV tree is quickly reconfigured to avoid the failed node.
The following changes are thus required:
IV trees are currently created per pool. A new IV tree for the system will thus be required. This tree can be used to propagate system attributes
A new ULT must be started in the mgmt service to regularly issue iv broadcast to update the deadline.
The pool service should no longer listen to SWIM events and instead be notified by the management service when a node is excluded by SWIM. This allows the pool service to accurately determine when to stop delaying writes based on the time when the management service made the exclusion.
The system map should be broadcasted along with the lease update.
The logic to try to refresh the least when the lease deadline has passed must be implemented on the engine.
The engine should return a special error code to the client once the lease deadline + lease time has passed. Client should try to refresh the pool map when getting such an error code and resubmit the RPC.
Surviving engines must delay write processing until the lease time has passed after exclusion.
...