We have an automated test infrastructure for building and running DAOS tests on Frontera.

Please see below for instructions on building and running tests.

Good Conduct

Before doing any work on Frontera, you should read and understand Good Conduct on Frontera.

You should also be aware of the limited credits for running jobs. After logging in, you can run:

Initial Setup

All of these setup instructions should be ran on a login node (E.g. login3.frontera).

Add your local binary directory to your PATH

Add the following line to ~/.bashrc. There should be an if block labeled "SECTION 2" where you should put this.

Setup directories and clone the test scripts

Install python dependencies for post-job scripts

These packages are for some of the .py scripts for post-processing results

Build MPI packages - Optional - Not Recommended

Build DAOS

Edit run_build.sh:

Configure these lines:

Optionally, you can choose to build a specific branch, commit, or cherry pick.

When executed on a login node, run_build.sh will only use a single process. It is recommended to build on a development node. This will build DAOS in <BUILD_DIR>/<date>/daos and build the latest hpc/ior.

Running Tests

The test script should be executed on a login node, which uses slurm to reserve nodes and run jobs.

Configure run_testlist.py:

Configure DAOS Configs

The DAOS configs are daos_scaled_testing/frontera/daos_{server,agent,control}.yml.

The client / test runner environment is defined in daos_scaled_testing/frontera/env_daos.

Test Variants

Test configs are in the daos_scaled_testing/frontera/tests directory. Each config is a python file that defines a list of test variants to run. For example, in tests/sanity/ior.py:

These parameters can be configured as desired. Only tests with 'enabled': True will run. To execute some tests:

You can then monitor the status of your jobs in the queue:

Get test results

When all the sbatch jobs complete, you should see the results at <RES_DIR>/<date>. You can extract the IOR and MdTest results into CSV format by running:

Storing and retrieving tests in the database

This is currently a work in progress. See https://github.com/daos-stack/daos_scaled_testing/blob/master/database/README.md for general usage.

Description of test cases

Frontera Test Plan

Validation Suite Workflow

These instructions cover running the Frontera performance suite for validation of Test Builds, Release Candidates, and performance-sensitive features.

Clone Test Scripts

Using a fresh clone of the test scripts keeps the configuration isolated from other ongoing work. E.g. cloning into a directory with the ticket number.

Build DAOS

Common Reasons for Build Failures

Merge conflict in *control*

DAOS requires sudo/root permission, but we do not have sudo/root on Frontera. In run_build.sh:merge_extra_daos_branches(), we merge a “hack” branch for DAOS to compile and run on Frontera. Using the latest branch as a base, we need to create a new branch, rebase on the conflicting commit in DAOS, resolve conflicts, push the branch, and update run_build.sh.

For example, the branch dbohning/io500-base-e2a10d7 is based on master commit e2a10d7, and has 5 “hacks”. Coordinate with @Dalton Bohning as necessary if a new conflict arises.

Run Sanity Tests

Before executing hundreds of test cases, make sure the test scripts and DAOS are behaving as expected. For example, sometimes a simple daos or dmg interface change can cause every test to fail.

After the tests complete, sanity check the results. All tests should contain Pass.

Optionally, you could look through one or more logs to check for unexpected warnings or errors.

Run Validation Suite

RF0 and EC Tests

There are currently 48 RF0, 42 EC IOR variants, and 42 EC MDTest variants

Rebuild Tests

There is currently 1 variant in this group.

It’s helpful to use a different results directory from the previous tests since these results aren’t stored in the database.

Max Scale Tests

There is currently 1 variant in this group.

It’s helpful to use a different results directory from the previous tests since these results aren’t stored in the database.

TCP Tests

There is currently 1 variant in this group.

It’s helpful to use a different results directory from the previous tests since these results aren’t stored in the database.

Gather Results

The results can be gathered and the CSVs optionally emailed to avoid SCP:

It’s not necessary to email the other variants, since they aren’t inserted into the database.

Common Reasons for Test Failures

clock drift is too high

The test framework sanity checks that the nodes are in-sync. This is usually transient, and re-running the job is the easiest solution. If this occurs very frequently, TACC might need to be notified.

out of memory

Frontera compute nodes only have ~85G RAM each. Most tests run with at most 60 seconds of write. Even with dedup and a special IOR patch, it’s possible to run out of space. If reducing the ior/mdtest stonewall time helps, then it’s probably just a space constraint.

Reporting

Navigate to Scripts

Insert to Database

From frontera_scaled_testing/database:

You should see:

Archive the Raw CSVs

Archive the raw CSV data in the SharePoint archive.

Generate Reports

There are two standard reports, which we currently base on v2.0.0. It’s best to keep using this baseline so there is a common reference point.

--base-commit is the hash for the version you want to compare against. I.e. the baseline.

--commit is the hash for the version you just ran with.

Upload Reports

The generated spreadsheets should be uploaded here. Create a new folder, following the existing naming convention.

Analyze Reports

Each sheet of each report has a % column for each metric on the far right. The percent difference is for base-commit -> commit. At the bottom is min, max, and mean for each column. Anything less than -5% is colored red. These are “tentative regressions”. If an entire column - i.e. all variants - of a sheet/group shows a regression, it’s more likely a “real regression”. If only some rows show regressions, it might just be variance with DAOS on Frontera. Re-run each of these “tentative regressions”. If performance is on-par with the baseline, it was most likely due to system variance. If performance is still low, it might be a “real regression”. Create a ticket for “real regressions”. Replace “tentative regressions” with the “better” result.

Replacing Results

1. Reverse the import by deleting:

2. Delete the “bad” job results

3. Re-generate the CSVs with get_results.py

4. Re-archive the CSVs if necessary

5. Re-import with db.py import

6. Re-generate the reports with db.py report

7. Replace the spreadsheets in SharePoint

Analyze Reports Again

The second analysis is similar to the first, except we don’t want to just keep running “tentative regressions”. If a regression is there after multiple runs, it’s likely a “real regression”. Deciding when results are just variance is mostly quantitative, but somewhat of an art.

Create Tickets

Create tickets for regressions and be sure to include or link to the logs.

Archive Logs

All job logs should be archived. There isn’t a centralized location, so put them somewhere stable on wolf, set permissions with chmod -R go+rx, and put the path on the ticket.