CI Functional Test Stages

What are Functional Test Stages?

A functional test stage is a CI stage in which functional tests are run. Functional tests are broken into different functional test stages based upon the cluster size and unique software or hardware features required to run the tests. The MD on SSD hardware stages will run a test with a MD on SSD specific server storage configuration. To support this feature tests must have storage: auto defined as part of their server configuration in their test yaml file.

The following table lists the current Functional Test Stages used in CI:

¹ either EL 8, EL 9, or Leap 15

² in practice this ends up being either ofi+verbs;ofi_rxm or ofi+tcp depending on the cluster selected [see Provider Settings and Test Stages for additional information]

³ either NVMe or VMD storage will be used depending on the cluster selected

When Functional Test Stages Are Run?

Functional test stages are run near the end of the PR after passing build and unit test stages. It can be time consuming to run all of the functional test stages for every pull request (PR), so only certain stages are run by default. Disabled stages can be enabled in a PR by specifying specific commit pragmas in the PR commit message. Note: commit pragmas must be defined in the most recent commit message in order for them to have an effect on the current push. The following table lists which functional test stages are run by default in a PR and the commit pragmas that can be used to reverse the default behavior:

What Functional Test Stages Need to be Run For a PR?

Normally the default enabled functional test stages are sufficient for PR testing, however there may be situations where additional stages need to be enabled:

• If your PR affects or requires PMEM you should enable the Functional Hardware Medium, Functional Hardware Medium Verbs Provider, and Functional Hardware Large stages

• If your PR affects or requires the UCX provider you should enable the Functional Hardware Medium UCX Provider stage.

• If your PR affects or requires VMD storage you should enable the Functional Hardware Medium VMD stage.

• If your PR affects or requires either EL 9 and/or Leap 15.6 you should enable the Functional on EL 9 and/or Functional on Leap 15.6 stage(s).
  
  

How Test Tags Can Skip a Functional Test Stage

In addition to the default run/skip behavior of a Functional Test Stage and defining a Skip-func-<stage>: true commit pragma in the PR commit message the use of the Test-tag: <tags> commit pragma can also result in a functional test stage being skipped. Each functional test stage is defined with a set of stage tags (as listed in the first table). At the start of each functional test stage a step - called Get test list - is run to determine if there are any tests that match the tags for the PR combined with the stage tags. If no tests are found matching the tags, then the stage is skipped because there are no tests for it to run. For example, if the commit message defined Test-tag: IorSmall and none of the functional tests with the IorSmall tag included the hw and large tags used with the Functional Hardware Large stage, then the stage would be skipped.

Provider Settings and Test Stages

Functional hardware test stages that do not define a specific provider - a.k.a. hardware stages without Provider in their name - do support specifying a specific provider through a commit pragma. For example, one can ensure that tests run in the Functional Hardware Medium stage run with the ofi+tcp provider by specifying the Test-provider-hw-medium: ofi+tcp commit pragma in their PR’s most recent commit message. Here’s a list of stages for which the provider can be manipulated with commit pragmas:

Additional References

• Commit Pragmas - DAOS Community - Confluence

• Test Tags - DAOS Community - Confluence