Test Tags

What Are Test Tags?

Functional tests use Avocado, which uses tags to label tests. A test can have many tags, and a tag can belong to many tests. These tags are specified in a docstring comment for each test function. For example:

def test_evict(self):
  """
  :avocado: tags=all,pr,daily_regression
  :avocado: tags=vm
  :avocado: tags=pool,pool_evict
  :avocado: tags=pool_evict_basic
  """

How Does Avocado Handle Sets of Tags?

Avocado uses commas and spaces to specify sets of tests. For example:

pr,ior - All tests tagged with both pr and ior
pr ior - All tests tagged with pr, plus all tests tagged with ior
pr,ior pr,mdtest - All tests tagged with both pr and ior, plus all tests tagged with both pr and mdtest

Avocado uses a hyphen to mean “not”. For example:

pr,-ior - All tests tagged with pr but not ior
pr,-ior,-mdtest - All tests tagged with pr but not ior or mdtest

More information about Avocado tag filtering can be found here.

How Do I Use Test Tags in a PR?

The /wiki/spaces/CI/pages/1389759093 specify which tests are run for a PR. By default, all tests tagged with pr are ran. I.e. Test-tag: pr. To run specific tests, you should first identify which tags you need. See Feature Tags for a list of common tags. To run a specific test case, you will need to find its tag in the test file. There are two pragmas to specify which tags to run:

Test-tag - Runs only the tags specified.
1. Test-tag: pr checksum - Runs pr tests and checksum tests.
2. Test-tag: checksum - Runs only checksum tests.
Features - Runs the default set of tags (pr), plus the tags specified minus full_regression.
1. Features: checksum - Equivalent to Test-tag: pr checksum,-full_regression

Whether to use Test-tag or Features mostly depends on whether you want to include the full_regression tests for the specified feature tag. In the future, full_regression tests will not be excluded, and Features will be recommended for most cases.

pr tests are almost always required to pass for a PR to be merged.

When Do I Need to Specify Test Tags?

When a change is 100% covered by existing unit tests, there is no need to specify additional tags.
When a change is covered by new unit tests, you might need to specify the corresponding Feature Tag.
When a change is not 100% covered by unit tests, you do need to specify the corresponding Feature Tag.
1. E.g. Middleware applications, high-level functional verification, 3rd-party libraries, integration
When a change modifies or adds Functional Tests or the Functional Test Framework, you do need to specify the corresponding Feature Tags and/or Specific Test Tags.
When a change fixes a failing Functional Test, you do need to specify the corresponding Specific Test Tags.

List of Frequency Tags

These tags determine when a test runs. Some tests might be ran at multiple frequencies.

Avocado Tag	Description
`pr`	Ran in CI for every PR. These are generally small core tests.
`daily_regression`	Ran in CI once daily. These are generally less critical than `pr` tests.
`full_regression`	Ran in CI weekly. These are generally less critical than `daily_regression`, or longer-running tests. These usually do not overlap with `daily_regression` tests.
`manual`	Manual tests for special purposes. These tests might: not run in CI, require special configuration, be for specific environments.
`soak`	SOAK tests are ran manually at specified intervals.
`deployment`	Ran when deploying a new DAOS installation.

List of Stage/Environment Tags

These tags determine where a test runs. Each test should only have one set of these tags.

Avocado Tag	Description
`hw,small`	Tests that are ran on the Small Hardware Functional Test stage
`hw,medium,ib2`	Tests that are ran on the Medium Hardware Functional Test stage.
`hw,large`	Tests that are ran on the Large Hardware Functional Test stage.
`vm`	Tests that are ran on the VM Functional Test stages. This tag is usually not specified explicitly. Instead, the lack of a `hw` tag represents VM tests.

List of Feature Tags

These tags group tests by common Features. This list is currently incomplete.

Avocado Tag	Description
`checksum`	Verifies `checksum` features.
`container`	Verifies `container` features.
`control`	Verifies `control`plane features.
`datamover`	Verifies Data Mover utilities. Additional sub-tags: `fs_copy` - `daos filesystem copy` tests `cont_clone` - `daos container clone` tests `dcp` - `mpifileutils/dcp` (distributed copy) tests `dsync` - `mpifileutils/dsync` (distributed sync) tests `dserialize` - `mpifileutils/daos-serialize` and `mpifileutils/daos-deserialize` (distributed serialization) tests
`daosperf`	Verifies `daos_perf`.
`deployment`	Verifies DAOS on a newly deployed system.
`dfuse`	Verifies `dfuse`.
`ec`	Verifies `ec` features.
`ior`	Verifies `ior`.
`mdtest`	Verifies `mdtest`.
`osa`	Verifies online and offline `osa` features.
`pool`	Verifies features related to `pool` operations.
`security`	Verifies `security` features such as ACLs, pool and container permissions.
`snap`	Verifies `snapshot` features.
`telemetry`	Verifies `telemetry` features.
`tx`	Verifies `transaction` features.

Finding Specific Test Tags

Each test has a unique tag that can be used to run only that test. When there is need to run a specific test, the tag must be looked up in the corresponding ftest/<dir>/<test_file>.py. Unique test tags are usually on the last line of Avocado tags. For example, from ftest/pool/evict.py, the pool_evict_basic tag will run only this test:

def test_evict(self):
  """
  :avocado: tags=all,pr,daily_regression
  :avocado: tags=vm
  :avocado: tags=pool,pool_evict
  :avocado: tags=pool_evict_basic
  """