ORB5 tests

Although Jenkins is designed for unit testing, ORB5 tests behave a bit differently. Indeed, it would take ages to write separate tests for each part of the code and we don't have the man power for such task.

Therefore, we decided to run the full code with different inputs to put pressure on modules one after the other. We have light test cases that can be run within few seconds/minutes on a single compute node, and heavier test cases which take some minutes on tens/hundreds of nodes. Light test cases are not physically relevant, as they have very poor particle and mesh resolutions. Heavy test cases check that the code can be used for production.

Each test contains a set of simulations to run and a set of comparisons to perform on the result files. Each simulation can pass or crash, and each comparison can detect significant differences between two results files or not.

The exhaustive list of tests can be found in the Jenkinsfile located at ORB5's root. It contains stand-alone tests for any commit (such as parallelisation or algorithmic tests), and a reproducibility test to detect modifications in the output file compared to another commit.

Jenkins interface

Note: As Jenkins interface is continuously evolving, some of the features described in this section might be obsolete.

Stage view

The stage view gives an overview of the main pipeline stages of last builds. Logs can be accessed by clicking on those stages. Stages will be described in Pipeline content section.

Pipeline view

Blue ocean interface (accessible from the left panel) gives a nicer overview of the pipeline stages by displaying parallel stages. It also offers easier browsing of the pipeline steps in case of error occurence.

Build view

By clicking on any build, you have access to the following information:

Artifacts is an expandable tree of the html reports of simulation differences detected by h5diff tool.
Changes lists new git commits since last build.
Test Result lists the failed tests (simulations or comparisons) with links to details.

Test result view

The test result view gathers information about crashing simulations and differences in output files. It shows the new failing tests and the fixed ones, as well as the number of commits since a failed test is failing.

For crashing simulations, you can expand standard ouptut and standard error to get an inside of what happened.

Build triggers

Automatic

A build is automatically triggered for each last commit of a push on the master branch.

Manual

In case the automatic build fails due to machine failure (e.g. slurm overloaded, disk full, ...), one can manually relaunch the build with the Build with parameters button on the left panel.

This method offers two optional parameters for specific needs:

Tested commit: Test a different commit from the latest master one. This option can be used to test old commits or feature branches.
Reproducibility reference commit: Take a specific commit as reference for reproducibility test instead of most recent succesful master simulation. This option can be used to test bug fixes or feature branches. The pipeline is a bit longer with this option because it has to re-compile and re-run references.

Pipeline content

SSH connexion to clusters

Simulations are run on CSCS machines. Jenkins instance owns an SSH key to connect to ohana@ela.cscs.ch during the initialization phase. The code is checked out on this cluster. (If a reproducibility reference commit has been given in input, it is also checked out.)

Make code versions

ORB5 binaries are compiled (in parallel) for different compilers and flags, on different CSCS clusters.

Run simulations

Simulations are submitted to slurm and job ids are stored. The pipeline waits until all of them are completed or failed.

Check results

Simulation outputs are compared with h5diff tool.

Save results

If the tested commit is the most recent one of master branch, the simulation outputs are stored for next builds.

Generate reports

Finally, two reports are generated:

junit report is an xml report sent to Jenkins for interface browsing (Test result view).
markdown report is a text report sent to Diffusion as a commit comment. The commit author will also receive this report by email.

Table of Contents