Advanced usage¶

Custom Runnable Identifier¶

In some cases, you might have a wrapper as an entry point for the tests, so Avocado will use only the wrapper as test id. For instance, imagine a Makefile with some targets (‘foo’, ‘bar’) and each target is one test. Having a single test suite with a test calling foo, it will make Avocado print something like this:

``` JOB ID : b6e5bdf2c891382bbde7f24e906a168af351154a JOB LOG : ~/avocado/job-results/job-2021-09-24T17.39-b6e5bdf/job.log

(1/1) make: STARTED (1/1) make: PASS (2.72 s)

This is happening because Avocado is using the ‘uri’ as identifier with in the current Runnables.

You can change that by setting a custom format with the option runner.identifier_format in you avocado.conf file. For instance:

` [runner] identifier_format = "{uri}-{args[0]}" `

With the above adjustment, running the same suite it will produce something like this:

``` JOB ID : 577b70b079e9a6f325ff3e73fd9b93f80ee7f221 JOB LOG : /home/local/avocado/job-results/job-2021-11-23T13.12-577b70b/job.log

(1/1) “/usr/bin/make-foo”: STARTED (1/1) “/usr/bin/make-foo”: PASS (0.01 s)

For the identifier_format you can use any f-string that it will use {uri}, {args} or {kwargs}. By default it will use {uri}.

When using args, since it is a list, you can use in two different ways: “{args}” for the entire list, or “{args[n]}” for a specific element inside this list. The same is valid when using “{kwargs}”. With kwargs, since it is a dictionary, you have to specify a key as index and then the values are used.

For instance if you have a kwargs value named ‘DEBUG’, a valid usage could be: “{kwargs[DEBUG]}” and this will print the current value to this variable (i.e: True or False).

Note

Please, keep in mind this is an experimental feature, and for now you have to use it in combination with documentation.

Note

Also, be aware this feature it is meant to set custom Runnable identifiers strings only.

Test Runner Selection¶

To effectively run a job with tests, Avocado makes use of a well described and pluggable interface. This means that users can choose (and developers can write) their own runners.

Runner choices can be seen by running avocado plugins:

...
Plugins that run test suites on a job (suite.runner):
nrunner nrunner based implementation of job compliant runner

And to select a different test runner (if another one exists):

avocado run --suite-runner=other_runner_plugin ...

Running tests with an external runner¶

It’s pretty standard to have organically grown test suites in most software projects, and these usually include a custom-built, specific test runner who knows how to find and run their tests.

Still, running those tests inside Avocado may be a good idea for various reasons, including having results in different human and machine-readable formats and collecting system information alongside those tests (the Avocado’s Sysinfo functionality), and more.

Avocado makes that possible using its “external runner” feature. The most basic way of using it is:

$ avocado-external-runner external_runner foo bar baz

In this example, Avocado will report individual test results for tests foo, bar, and baz. The actual results will be based on the return code of individual executions of /path/to/external_runner foo, /path/to/external_runner bar and finally /path/to/external_runner baz.

As another way to explain how this feature works, think of the “external runner” as an interpreter. The individual tests as anything that this interpreter recognizes and can execute. A UNIX shell, say /bin/sh could be considered an external runner, and files with shellcode could be viewed as tests:

$ echo "exit 1" > /tmp/fail
$ echo "exit 0" > /tmp/pass

$ avocado-external-runner /bin/sh /tmp/pass /tmp/fail
JOB ID     : 874cab7e2639f1e2244246c69a5e0d3e1afefee0
JOB LOG    : ~/avocado/job-results/job-2022-01-19T15.33-874cab7/job.log
 (external-runner-2/2) /bin/sh-/tmp/fail: STARTED
 (external-runner-1/2) /bin/sh-/tmp/pass: STARTED
 (external-runner-2/2) /bin/sh-/tmp/fail: FAIL (0.01 s)
 (external-runner-1/2) /bin/sh-/tmp/pass: PASS (0.01 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : ~/avocado/job-results/job-2022-01-19T15.33-874cab7/results.html
JOB TIME   : 1.10 s

Note

This example is pretty obvious and could be achieved by giving /tmp/pass and /tmp/fail shell “shebangs” (#!/bin/sh), making them executable (chmod +x /tmp/pass /tmp/fail), and running them as “SIMPLE” tests.

But now consider the following example:

$ avocado-external-runner curl redhat.com "google.com -v"
JOB ID     : fa68dd49a4c00e5a3c2e0fe45c6b3b0ed1b6495e
JOB LOG    : ~/avocado/job-results/job-2022-01-19T15.37-fa68dd4/job.log
 (external-runner-2/2) /bin/curl-google.com: STARTED
 (external-runner-1/2) /bin/curl-redhat.com: STARTED
 (external-runner-2/2) /bin/curl-google.com: PASS (0.28 s)
 (external-runner-1/2) /bin/curl-redhat.com: PASS (5.39 s)
RESULTS    : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : ~/avocado/job-results/job-2022-01-19T15.37-fa68dd4/results.html
JOB TIME   : 6.38 s

This effectively makes /bin/curl an “external test runner”, responsible for trying to fetch those URLs, and reporting PASS or FAIL for each of them.