...

The current layout version for a pool should be shown via pool properties. It is an integer number that is not directly related to the DAOS version number. A pool can be upgraded to the latest supported version via “dmg pool upgrade”. A pool is inaccessible during the upgrade. The upgrade can complete promptly if the layout changes were minor (e.g. adding new properties like in the case of 2.0 to 2.2) or be long running (e.g. placement changes). The dmg pool upgrade returns immediately and status is reported asynchronously via another pool property (similar to the container state property and ). The pool status can be reused for catastrophic recovery too )and should probably be a bit flag if we want to record multiple state in the future.

If the system fails or is shutdown during the an pool upgrade, the process will be automatically resumed once the engine(s) are restarted. The status of the pool will be marked as upgrade in progress.

...

Primary interface will be via dmg pool upgrade.

$ dmg pool get-prop mypool
[…]
Pool version: 1
Pool status: Online

$ dmg pool upgrade mypool
Upgrading mypool from version 1 to version 2

$ dmg pool get-prop mypool
[…]
Pool version: 1
Pool status: Upgrading

$ dmg pool get-prop mypool
[…]
Pool version: 2
Pool status: Online

Some metrics might be exported in the future to monitor the progress of the upgrade.

...

How the feature will be tested? Unit tests, functional tests and system tests need to be covered.
Describe the extra soak/performance tests that should be added.

A bunch of functional tests need to be integrated into the CI:

• verify that data integrity is preserved after an upgrade

• verify that an upgrade is effective (new properties are added in the case of 2.2)

• verify that a pool cannot be accessed while an upgrade is in progress

• verify that an interrupted upgrade (e.g. system shutdown or failure) is resumed upon restart

• verify that a new container created in a pool with a old version does not enable the new features

In addition to this, end-to-end upgrade/downgrade tests should be developed.

Project Milestones

Description of the different milestones delivering incremental functionality.
Describe what will work/not work and what will be validated.
Targeted date for each milestone.

• Milestone 1:

  • define the infrastructure to initiate a pool upgrade including

    • dmg interface

    • versioning scheme with new associated pool properties

    • revoke open handle and check that no rebuild/reint/extend in progress

    • restart after shutdown/failure

  • new container created in a pool with a old version should not enable the new features.

  • manage upgrade from 2.0 to 2.2

    • add new pool/container properties added between 2.0 and 2.2

  • downgrading the DAOS software of an upgraded pool report a proper RAS error.

• Milestone 2:

  • address use case for data migration between 2.2 and 2.4 with the EC placement change.

  • add metrics to report progress

Gantt Chart

Provide Gantt chart for each milestone.

...