Option to change redundancy level from engine to server

Patch landed, closing the ticket.

thanks for the testing, if find bug later please feel free to report.

Thanks for the info. (I remember the rebuild result doesn’t show the redundancy capability.) I tested the following with 3 nodes with 2 ranks/node.

rf_lvl:2,rf:1 - Used RP_2G1

• Bring down two consecutive ranks, 4 and 5, which are in the same node (wolf-121): Container Health remains HEALTHY.

• Bring down two ranks in two different nodes, 3 (wolf-120) and 5 (wolf-121): Container Health becomes UNCLEAN.

rf_lvl:1,rf:1 - Used RP_2GX

• Bring down two consecutive ranks, 4 and 5, which are in the same node (wolf-121): Container Health becomes UNCLEAN.

It looks like the feature is working as expected.

I’ll try RP_2GX with your patch later. Thanks.

for the problem of “I get an error if I try to use RP_2GX with rf_lvl:2” could you please test if the problem still occurs with this WIP patch?

“I stopped ranks from two different nodes, which exceeds the rf:1 limit based on your explanation”

yes it should break the RF:1, for this case should be able to see RF broken err msg in the log? and the opened container should not be able to write/read after that right?

But for the rebuilding of the RP_2G1 obj, it does not mean it has to fail, because as long as there is one replica remaining and if the rebuild can find alive spare target the rebuild can work (rebuild’s IO is allowed even when RF broken).

Just a note again that before resolved, current pool map with the info of rank0-1 belong to NODE0, rank2-3 belong to NODE1. So current test should also based on this assumption.

Thanks for the explanation. I’m doing some quick testing now and have some questions.

I’m using 3 nodes with 2 engines/node. I’m creating a container with rf_lvl:2,rf:1 across the 6 ranks. Then I run IOR with RP_2G1 oclass. At this point, if I stop three ranks where two of them are in the same node with dmg system stop, then the subsequent rebuild will succeed (will be done). I stopped ranks from two different nodes, which exceeds the rf:1 limit based on your explanation. Is this expected?

Also, I get an error if I try to use RP_2GX with rf_lvl:2. I see this with some other X oclasses too.

if the container is created with rf_lvl:2 (NODE), then with in one NODE if any engine/vos_target (despite of one or more) failed it will be treated as one NODE failure.

If the case you mentioned, stop one rank from wolf-1 and one rank from wolf-2, that will cause both NODEs failed. So for the container created with RF:1 it will report RF broken as #node_failure > rf, and for the container created with RF:2 will not cause RF broken.

But a related problem is now the topology of which engine belong to which NODE/server was not parsed/passed correctly by control plane, it need to be addressed by , before that being addressed, currently the pool map with info of engine 0:1 belong to NODE/server_0, and engine 2:3 belong to NODE/server_1 and so on, that possibly does not match with real config. So based on current master’s behavior, if stop rank2 and rank3 it will be treated as ONE NODE failure (if the container created with rf_lvl:2) although possibly in real configuration the rank2 and rank 3 possibly be on different NODEs.