Test Tags

1 What Are Test Tags?
2 How Does Avocado Handle Sets of Tags?
3 How Do I Use Test Tags in a PR?
4 When Do I Need to Specify Test Tags?
5 List of Frequency Tags
6 List of Stage/Environment Tags
7 List of Feature Tags
8 Finding Specific Test Tags
- 8.1 Short Answer
- 8.2 Long Answer

What Are Test Tags?

Functional tests use Avocado, which uses tags to label tests. A test can have many tags, and a tag can belong to many tests. These tags are specified in a docstring comment for each test function. For example:

def test_create_max_pool(self):
  """
  :avocado: tags=all,daily_regression                  # when to run - see frequency tags below
  :avocado: tags=hw,medium                             # where to run - see stage/environment tags below
  :avocado: tags=pool                                  # features - see feature tags below
  :avocado: tags=PoolCreateTests,test_create_max_pool  # test class and unique test method tags
  """

How Does Avocado Handle Sets of Tags?

Avocado uses commas and spaces to specify sets of tests. For example:

pr,ior - All tests tagged with both pr and ior
pr ior - All tests tagged with pr, plus all tests tagged with ior
pr,ior pr,mdtest - All tests tagged with both pr and ior, plus all tests tagged with both pr and mdtest

Avocado uses a hyphen to mean “not”. For example:

pr,-ior - All tests tagged with pr but not ior
pr,-ior,-mdtest - All tests tagged with pr but not ior or mdtest

More information about Avocado tag filtering can be found here.

How Do I Use Test Tags in a PR?

The Test-tag and Features commit pragmas specify which tests are run for a PR. By default, all tests tagged with pr are ran. I.e. Test-tag: pr. To run specific tests, you should first identify which tags you need. See Feature Tags for a list of common tags. To run a specific test case, you will need to find its tag in the test file. There are two pragmas to specify which tags to run:

Features - recommended - Runs the default set of tags (pr), plus the tags specified.
1. Features: checksum - Equivalent to Test-tag: pr checksum
Test-tag - advanced - Runs only the tags specified.
1. Test-tag: pr checksum - Runs pr tests and checksum tests.
2. Test-tag: checksum - Runs only checksum tests.

pr tests are almost always required to pass for a PR to be merged. The Features tag should be used in most cases.

When Do I Need to Specify Test Tags?

You should specify one or more Feature Tags when a change is not 100% covered by unit tests.
You should to specify the corresponding Feature Tags and/or Specific Test Tags when modifying Functional Tests, the Functional Test Framework, or fixing a failing test.

List of Frequency Tags

These tags determine when a test runs. Some tests might be ran at multiple frequencies.

Avocado Tag	Description

Avocado Tag	Description
`pr`	Ran in CI for every PR. These are generally small core tests.
`daily_regression`	Ran in CI once daily. These are generally less critical than `pr` tests.
`full_regression`	Ran in CI weekly. These are generally less critical than `daily_regression`, or longer-running tests. These usually do not overlap with `daily_regression` tests.
`manual`	Manual tests for special purposes. These tests might: not run in CI, require special configuration, be for specific environments.
`soak`	SOAK tests are ran manually at specified intervals.
`deployment`	Ran when deploying a new DAOS installation.

List of Stage/Environment Tags

These tags determine where a test runs. Each test should only have one set of these tags.

Avocado Tag	Description

Avocado Tag	Description
`hw,medium`	Tests that are ran on the Functional Hardware Medium stage.
`hw,large`	Tests that are ran on the Functional Hardware Large stage.
`vm`	Tests that are ran on the VM Functional on [CentOS, Leap, Ubuntu] stages. This tag is usually not specified explicitly. Instead, the lack of a `hw` tag represents VM tests.

List of Feature Tags

These tags group tests by common Features. The list recorded here is a starting point, and is likely to be incomplete or out-of-date. When evaluating which tag(s) should be used in the Features: pragma, remember that the tests themselves define the available set of tags, and can be consulted for hints as to which tags are likely to provide the best coverage for the PR.

Avocado Tag	Description

Avocado Tag	Description
`aggregation`	Verifies `aggregation`-level features.
`cart`	Verifies `cart`-level features.
`checksum`	Verifies `checksum` features.
`container`	Verifies `container` features.
`control`	Verifies `control`plane features.
`datamover`	Verifies Data Mover utilities. Additional sub-tags: `mfu` - mpifileutils `mfu_dcp` - mpifileutils/dcp `mfu_dsync` - mpifileutils/dsync `mfu_daos_serialize` - mpifileutils/daos-serialize `mfu_daos_deserialize` - mpifileutils/daos-deserialize `daos_cont_clone` - daos container clone `daos_fs_copy` - daos filesystem copy
`daos_test`	The `daos_test` suite.
`daosperf`	Verifies `daos_perf`.
`deployment`	Verifies DAOS on a newly deployed system.
`dfuse`	Verifies `dfuse`.
`dfs`	Verifies DFS. incomplete
`dmg`	Verifies `dmg` functionality.
`ec`	Verifies `ec` features.
`hdf5`	Verifies HDF5. incomplete
`ior`	Verifies `ior`.
`pil4dfs`	Verifies the `pil4dfs` library.
`mdtest`	Verifies `mdtest`.
`object`	Verifies `object`-level features.
`osa`	Verifies online and offline `osa` features.
`pool`	Verifies features related to `pool` operations.
`rebuild`	Verifies `rebuild`.
`recovery`	Verifies `ddb` and catastrophic recovery.
`cat_recov`	Verifies catastrophic recovery.
`security`	Verifies `security` features such as ACLs, pool and container permissions.
`server`	Verifies `server`-level features.
`snap`	Verifies `snapshot` features.
`telemetry`	Verifies `telemetry` features.
`tx`	Verifies `transaction` features.

Finding Specific Test Tags

Short Answer

This helpful utility will tell you the minimal tags that represent all tests in a given file(s):

cd src/tests/ftest/
./tags.py list pool/create.py
PoolCreateTests

Long Answer

Each test is uniquely tagged with its method name and the test class it belongs to. For example, from ftest/pool/create.py:

test_create_max_pool will run only test_create_max_pool
test_create_no_space_loop will run only test_create_no_space_loop
PoolCreateTests will run all tests in this class (there is usually one class per file, but in rare cases there are multiple classes per file)

def test_create_max_pool(self):
"""
    :avocado: tags=all,daily_regression
    :avocado: tags=hw,medium
    :avocado: tags=pool
    :avocado: tags=PoolCreateTests,test_create_max_pool
"""

def test_create_no_space_loop(self):
"""
    :avocado: tags=all,pr,daily_regression
    :avocado: tags=hw,medium
    :avocado: tags=pool
    :avocado: tags=PoolCreateTests,test_create_no_space_loop
"""