Test Tags

What Are Test Tags?

Functional tests use Avocado, which uses tags to label tests. A test can have many tags, and a tag can belong to many tests. These tags are specified in a docstring comment for each test function. For example:

def test_create_max_pool(self):
  """
  :avocado: tags=all,daily_regression                  # when to run - see frequency tags below
  :avocado: tags=hw,medium                             # where to run - see stage/environment tags below
  :avocado: tags=pool                                  # features - see feature tags below
  :avocado: tags=PoolCreateTests,test_create_max_pool  # test class and unique test method tags
  """

How Does Avocado Handle Sets of Tags?

Avocado uses commas and spaces to specify sets of tests. For example:

pr,ior - All tests tagged with both pr and ior
pr ior - All tests tagged with pr, plus all tests tagged with ior
pr,ior pr,mdtest - All tests tagged with both pr and ior, plus all tests tagged with both pr and mdtest

Avocado uses a hyphen to mean “not”. For example:

pr,-ior - All tests tagged with pr but not ior
pr,-ior,-mdtest - All tests tagged with pr but not ior or mdtest

More information about Avocado tag filtering can be found here.

How Do I Use Test Tags in a PR?

The /wiki/spaces/CI/pages/1389759093 specify which tests are run for a PR. By default, all tests tagged with pr are ran. I.e. Test-tag: pr. To run specific tests, you should first identify which tags you need. See Feature Tags for a list of common tags. To run a specific test case, you will need to find its tag in the test file. There are two pragmas to specify which tags to run:

Features - recommended - Runs the default set of tags (pr), plus the tags specified.
1. Features: checksum - Equivalent to Test-tag: pr checksum
Test-tag - advanced - Runs only the tags specified.
1. Test-tag: pr checksum - Runs pr tests and checksum tests.
2. Test-tag: checksum - Runs only checksum tests.

pr tests are almost always required to pass for a PR to be merged. The Features tag should be used in most cases.

When Do I Need to Specify Test Tags?

You should specify one or more Feature Tags when a change is not 100% covered by unit tests.
You should to specify the corresponding Feature Tags and/or Specific Test Tags when modifying Functional Tests, the Functional Test Framework, or fixing a failing test.

List of Frequency Tags

These tags determine when a test runs. Some tests might be ran at multiple frequencies.

Avocado Tag	Description
`pr`	Ran in CI for every PR. These are generally small core tests.
`daily_regression`	Ran in CI once daily. These are generally less critical than `pr` tests.
`full_regression`	Ran in CI weekly. These are generally less critical than `daily_regression`, or longer-running tests. These usually do not overlap with `daily_regression` tests.
`manual`	Manual tests for special purposes. These tests might: not run in CI, require special configuration, be for specific environments.
`soak`	SOAK tests are ran manually at specified intervals.
`deployment`	Ran when deploying a new DAOS installation.

List of Stage/Environment Tags

These tags determine where a test runs. Each test should only have one set of these tags.

Avocado Tag	Description
`hw,small`	Tests that are ran on the Functional Hardware Small stage
`hw,medium`	Tests that are ran on the Functional Hardware Medium stage.
`hw,large`	Tests that are ran on the Functional Hardware Large stage.
`vm`	Tests that are ran on the VM Functional on [CentOS, Leap, Ubuntu] stages. This tag is usually not specified explicitly. Instead, the lack of a `hw` tag represents VM tests.

List of Feature Tags

These tags group tests by common Features. The list recorded here is a starting point, and is likely to be incomplete or out-of-date. When evaluating which tag(s) should be used in the Features: pragma, remember that the tests themselves define the available set of tags, and can be consulted for hints as to which tags are likely to provide the best coverage for the PR.

Avocado Tag	Description
`checksum`	Verifies `checksum` features.
`container`	Verifies `container` features.
`control`	Verifies `control`plane features.
`datamover`	Verifies Data Mover utilities. Additional sub-tags: `mfu` - mpifileutils `mfu_dcp` - mpifileutils/dcp `mfu_dsync` - mpifileutils/dsync `mfu_daos_serialize` - mpifileutils/daos-serialize `mfu_daos_deserialize` - mpifileutils/daos-deserialize `daos_cont_clone` - daos container clone `daos_fs_copy` - daos filesystem copy
`daosperf`	Verifies `daos_perf`.
`deployment`	Verifies DAOS on a newly deployed system.
`dfuse`	Verifies `dfuse`.
`dfs`	Verifies DFS. incomplete
`dmg`	Verifies `dmg` functionality.
`ec`	Verifies `ec` features.
`hdf5`	Verifies HDF5. incomplete
`ior`	Verifies `ior`.
`mdtest`	Verifies `mdtest`.
`osa`	Verifies online and offline `osa` features.
`pool`	Verifies features related to `pool` operations.
`security`	Verifies `security` features such as ACLs, pool and container permissions.
`snap`	Verifies `snapshot` features.
`telemetry`	Verifies `telemetry` features.
`tx`	Verifies `transaction` features.

Finding Specific Test Tags

We are in the process of using each test’s method name as the unique tag to run that test. This tag will be on the last line of the Avocado tags. For example, from ftest/pool/create.py, the test_create_max_pool tag will run only this test:

def test_create_max_pool(self):
"""...
    :avocado: tags=all,daily_regression
    :avocado: tags=hw,medium
    :avocado: tags=pool
    :avocado: tags=PoolCreateTests,test_create_max_pool
"""