This page tracks ideas for improving the stability and throughput of the DAOS CI system. This just supplements Jira for communication & progress reporting purposes.
All the individual improvement ideas are tracked in Jira epic DAOS-3929.
| Ticket(s) | Task | Status | Result |
|---|---|---|---|
DAOS-3914 CORCI-840 | Increase throughput on HW clusters by running multiple servers per HW node and split one of the HW clusters. | blocked | Wait for new HW. |
| CORCI-841 | Build new HW clusters and get them into CI as soon as HW shows up | blocked | HW ships 1/22 |
| Done | Splitting clusters from 8 to 4 node will not make an immediate substantial impact. This is going to help long term though, particularly when adding the new hardware. | ||
| CORCI-717 | Node failures are often the cause of intermittent CI issues but need console log to debug | todo | |
| Done | |||
| Done | |||
| CORCI-711 | HW nodes are provisioned from snapshot, fix out-of-space and other cruft issues that cause intermittent failures | in-progress | |
DAOS-3607
| Increase flexibility to run different groupings of tests by running from RPMs | in-progress | |
| CORCI-843 | If a change impacts files in the "doc" directory then skip unnecessary build/test cycles. | in-progress | |
| DAOS-3921 | Reduce wait time in daos_test rebuild subtests | in-progress | |
| Done | ~50 minute reduction? | ||
| DAOS-3930 | IorSmall runtimes are all over the map | todo | |
| Done | Reduced intermittent failures | ||
| Done | Modest improvement, but not what expected. | ||
| Investigate GitHub Checks as an alternative to the commit statuses that we currently use | todo | ||
| Investigate moving more error reporting out of Jenkins Workflow steps and into JUnit results | todo | ||
Data-mine Jenkins for statistics on how many PR commits are test retries as a metric of how often we are hitting intermittent failures
| todo | ||
Data-mine Jenkins for statistics on how many PR commits have actual patches to rectify failed tests
| todo |
The primary metric for tracking improvements is total hours master jobs take to complete. Total time includes waiting time plus run time. The chart below includes randomly select successful runs against master.
