Versions Compared

Version Old Version 19 New Version Current
Changes made by

Dalton Bohning

Dalton Bohning

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
languagepy
def test_evictcreate_max_pool(self):
  """
  :avocado: tags=all,pr,daily_regression                  # when to run - see frequency tags below
  :avocado: tags=vmhw,medium                             # where to run - see stage/environment tags below
  :avocado: tags=pool,pool_evict                                  # features - see feature tags below
  :avocado: tags=pool_evict_basicPoolCreateTests,test_create_max_pool  # test class and unique test method tags
  """

How Does Avocado Handle Sets of Tags?

Avocado uses commas and spaces to specify sets of tests. For example:

  1. pr,ior - All tests tagged with both pr and ior

  2. pr ior - All tests tagged with pr, plus all tests tagged with ior

  3. pr,ior pr,mdtest - All tests tagged with both pr and ior, plus all tests tagged with both pr and mdtest

Avocado uses a hyphen to mean “not”. For example:

  1. pr,-ior - All tests tagged with pr but not ior

  2. pr,-ior,-mdtest - All tests tagged with pr but not ior or mdtest

More information about Avocado tag filtering can be found here.

...

The /wiki/spaces/CI/pages/1389759093 specify which tests are run for a PR. By default, all tests tagged with pr are ran. I.e. Test-tag: pr. To run specific tests, you should first identify which tags you need. See Feature Tags for a list of common tags. To run a specific test case, you will need to find its tag in the test file. There are two pragmas to specify which tags to run:

...

  1. Features -

...

  1. recommended - Runs

...

  1. the

...

  1. Test-tag: pr checksum - Runs pr tests and checksum tests.

  2. Test-tag: checksum - Runs only checksum tests.

...

  1. default set of tags (pr), plus the tags specified

...

  1. .

    1. Features: checksum - Equivalent to Test-tag: pr checksum

...

...

  1. Test-tag

...

  1. - advanced - Runs only the tags specified.

    1. Test-tag: pr checksum - Runs pr tests and checksum tests.

    2. Test-tag: checksum - Runs only checksum tests.

Info

pr tests are almost always required to pass for a PR to be merged. The Features tag should be used in most cases.

When Do I Need to Specify Test Tags?

...

When a change is 100% covered by existing unit tests, there is no need to specify additional tags.

...

When a change is covered by new unit tests, you might need to specify the corresponding Feature Tag.

...

  • You should specify one or more Feature Tags when a change is not 100% covered by unit tests

...

  1. E.g. Middleware applications, high-level functional verification, 3rd-party libraries, integration

...

...

  • when modifying Functional Tests, the Functional Test Framework, or fixing a failing test.

List of Frequency Tags

These tags determine when a test runs. Some tests might be ran at multiple frequencies.

Avocado Tag

Description

pr

Ran in CI for every PR. These are generally small core tests.

daily_regression

Ran in CI once daily. These are generally less critical than pr tests.

full_regression

Ran in CI weekly. These are generally less critical than daily_regression, or longer-running tests. These usually do not overlap with daily_regression tests.

manual

Manual tests for special purposes. These tests might: not run in CI, require special configuration, be for specific environments.

soak

SOAK tests are ran manually at specified intervals.

deployment

Ran when deploying a new DAOS installation.

List of Stage/Environment Tags

These tags determine where a test runs. Each test should only have one set of these tags.

Avocado Tag

Description

hw,

small

Tests that are ran on the Functional Hardware Small stage

hw,

medium

,ib2

Tests that are ran on the Functional Hardware Medium stage.

hw,large

Tests that are ran on the Functional Hardware Large stage.

vm

Tests that are ran on the VM Functional on [CentOS, Leap, Ubuntu] stages. This tag is usually not specified explicitly. Instead, the lack of a hw tag represents VM tests.

Anchor
Feature-Tags
Feature-Tags
List of Feature Tags

These tags group tests by common Features. This The list is currently incomplete.recorded here is a starting point, and is likely to be incomplete or out-of-date. When evaluating which tag(s) should be used in the Features: pragma, remember that the tests themselves define the available set of tags, and can be consulted for hints as to which tags are likely to provide the best coverage for the PR.

Avocado Tag

Description

aggregation

Verifies aggregation-level features.

cart

Verifies cart-level features.

checksum

Verifies checksum features.

container

Verifies container features.

control

Verifies controlplane features.

datamover

Verifies Data Mover utilities.

Additional sub-tags:

  • fs_copy - daos filesystem copy tests

  • cont_clone - daos container clone tests

    • mfu - mpifileutils

      • mfu_dcp - mpifileutils/dcp

    (distributed copy) tests

      • mfu_dsync - mpifileutils/dsync

    (distributed sync) testsdserialize

      • mfu_daos_serialize - mpifileutils/daos-serialize

    and

      • mfu_daos_deserialize - mpifileutils/daos-deserialize

    (distributed serialization) tests

    • daos_cont_clone - daos container clone

    • daos_fs_copy - daos filesystem copy

    daos_test

    The daos_test suite.

    daosperf

    Verifies daos_perf.

    deployment

    Verifies DAOS on a newly deployed system.

    dfuse

    Verifies dfuse.

    dfs

    Verifies DFS. incomplete

    dmg

    Verifies dmg functionality.

    ec

    Verifies ec features.

    hdf5

    Verifies HDF5. incomplete

    ior

    Verifies ior.

    pil4dfs

    Verifies the pil4dfs library.

    mdtest

    Verifies mdtest.

    object

    Verifies object-level features.

    osa

    Verifies online and offline osa features.

    pool

    Verifies features related to pool operations.

    rebuild

    Verifies rebuild.

    recovery

    Verifies ddb and catastrophic recovery.

    cat_recov

    Verifies catastrophic recovery.

    security

    Verifies security features such as ACLs, pool and container permissions.

    server

    Verifies server-level features.

    snap

    Verifies snapshot features.

    telemetry

    Verifies telemetry features.

    tx

    Verifies transaction features.

    Anchor
    Specific-Test-Tags
    Specific-Test-Tags
    Finding Specific Test Tags

    ...

    Short Answer

    This helpful utility will tell you the minimal tags that represent all tests in a given file(s):

    Code Block
    languagebash
    cd src/tests/ftest/
./tags.py list pool/create.py
PoolCreateTests

    Long Answer

    Each test is uniquely tagged with its method name and the test class it belongs to. For example, from ftest/pool/evict.py, the pool_evict_basic tag will run only this test:create.py:

    • test_create_max_pool will run only test_create_max_pool

    • test_create_no_space_loop will run only test_create_no_space_loop

    • PoolCreateTests will run all tests in this class (there is usually one class per file, but in rare cases there are multiple classes per file)

    Code Block
    languagepy
    def test_create_max_evictpool(self):
"""
    :avocado: tags=all,daily_regression
    :avocado: tags=hw,medium
    :avocado: tags=pool
    :avocado: tags=PoolCreateTests,test_create_max_pool
"""

def test_create_no_space_loop(self):
"""
    :avocado: tags=all,pr,daily_regression
    :avocado: tags=vmhw,medium
    :avocado: tags=pool,pool_evict
    :avocado: tags=pool_evict_basic
 PoolCreateTests,test_create_no_space_loop
"""