Running Tests Locally

fwci run runs a workflow’s tests from your repository’s .firmwareci on your own machine. A test runs as it would in a job: the DUT’s pre-stage, the test’s stages, then the DUT’s post-stage, in the same container image. The run needs no login and doesn’t contact FirmwareCI, so nothing appears on the dashboard.

Use it to work on a test or a DUT’s stages before you push them, or to run tests in your own CI. With --remote or --synced, fwci run runs the tests on FirmwareCI instead, see Running a Job. Every flag, variable and exit code of a local run is listed in the Local Runs reference.

Prerequisites

  • The fwci CLI, see Installation. A local run doesn’t need fwci login.
  • Docker on the same machine. Rootless Docker works.
  • A repository with a .firmwareci directory, see Repository Setup.
  • The DUT, and for a DUT with reservation-system: dutctl its dutagent, reachable from your machine. The tests run here in your machine’s network and connect to the addresses in dut.yaml themselves, such as over SSH, which often works only through the lab’s VPN, see Network.
  • On Linux ARM64: the Linux x64 build of fwci as well, see The Runner.

Run a Test

Run fwci inside the repository. -w names the workflow, by its directory under .firmwareci/workflows or by the name in its workflow.yaml. -t picks a test by the name in its file. Binaries come last, as <key>=<file>:

$ fwci run -w crystal-core -t boot Binary=./build/coreboot.rom

The run takes a DUT from .firmwareci/duts whose label is the workflow’s runs-on: the first one that is free, in the order of their names. The run names the one it took by its directory, such as DUT dut-crystal-core (first free of 2 labeled crystal-core). --dut picks one, by its directory under .firmwareci/duts or by the name in its dut.yaml. It needs the workflow’s label too. If no DUT has the label, nothing runs:

No DUT in .firmwareci/duts is labeled "crystal-core", which workflow crystal-core runs on.
To run the tests on FirmwareCI instead, add --remote.

Each step shows as one line once it passed. While a step runs, one line shows its name, how long it has run and its newest output line, such as the serial console’s. A step that fails shows its whole log, and each test ends with the summary fwci view shows. --verbose shows every step’s whole log as it comes. Every step’s whole log also goes into one file for the run, under ~/.cache/fwci/logs/, and each test’s summary shows its path. --log <file> writes it to that file instead.

To run more than one test, repeat -t, as -t flash -t boot, or list the names, as -t flash,boot. The tests run one after the other in that order, on the same DUT, each with the DUT’s stages. Without -t, every enabled test of the workflow runs, in the order of their names; give -t when the order matters. Each test starts with a line like Test 2/3: boot, and the run ends with one line for all of them:

1 of 3 tests failed: boot (step "Wait for SSH service").

Before anything runs, fwci checks every test: its file, and the binaries, storage, secrets and keys it uses. If one can’t start, none runs. What the tests use and you didn’t give, fwci lists at once. A file or directory you gave that isn’t there, it names one at a time. A name the workflow has no test for stops the run too, and fwci lists the workflow’s tests.

The tests run in the image named in the workflow’s workflow.yaml. Without one, they run in the default job image, which needs no login (see Job Images). The first run pulls the image, which can take a few minutes.

Pass What a Job Gets From FirmwareCI

On the server, a job gets storage, secrets and SSH keys from FirmwareCI. A local run takes them from your machine:

A test usesOn the server it comes fromIn a local run
[[input.Binary]]the binary you uploadBinary=<file>
[[storage.tools]]the storage tools in storage.yaml--storage tools=<directory>
[[secrets.APIToken]]the workflow’s secretsthe environment variable FWCI_SECRET_APIToken
[[ssh-keys.lab]]your organization’s SSH keys--ssh-key lab=<private key file>
[[ssh-keys.default]]the default keybuilt into fwci
$ export FWCI_SECRET_APIToken=...
$ fwci run -w crystal-core -t boot \
    --storage tools=./tools \
    --ssh-key lab=$HOME/.ssh/lab_ed25519 \
    Binary=./build/coreboot.rom

Storage. The directory stands in for the files the storage gets from FirmwareCI. To start from the files you uploaded, get them with fwci files download, which needs a login. An entry with several files comes as a .tar.gz: unpack it into a directory and pass that. The storage’s commands and paths in storage.yaml apply as on the server. Each test gets its own copy of the directory, so what one test changes, the next one doesn’t see.

Secrets. The variable is named exactly like the secret, so [[secrets.APIToken]] reads FWCI_SECRET_APIToken. An empty variable counts as not set. A secret you didn’t set stops the run before anything runs, naming the variable. A shell can’t export a name with a -, so set such a secret with env:

$ env 'FWCI_SECRET_ANOTHER-TOKEN=...' fwci run -w crystal-core -t boot

SSH keys. --ssh-key takes the private key. Its public half is read from the same path with .pub, if that file exists. A test that names no key tries every key you pass and the default key. [[ssh-keys.default]] is the key pair the Test Images accept, and fwci puts it at /root/.ssh/fwci in the container, as a job does. If your FirmwareCI instance has its own default key, pass it with --ssh-key default=<file>.

When Someone Else Has the Board

For a DUT with reservation-system: dutctl, the run locks the device on its agent before the first test and unlocks it after the last one, so no job gets the device between two tests. It locks it as <user>@<host>/<run-ID>, so the agent’s device list shows whose run has it.

If someone else holds the device, the run stops with exit code 10 before any test runs. It names the DUT by the name in its dut.yaml, and says who holds it and when their lease ends at the latest:

Crystal-Core is held by FirmwareCI job 01M75HGNQDKHNZE80WA5R9PDP5, whose lease ends in 7h52m at the latest.
Run with --wait to wait until it's free, or with --wait=30m to wait at most 30 minutes.

When several DUTs have the workflow’s label, the run takes the first one that is free, and stops with exit code 10 only when someone holds each of them. It then names every holder.

A job usually ends long before its lease does. To wait for the device:

# Wait until the device is free
$ fwci run -w crystal-core -t boot --wait

# Wait at most 30 minutes, then exit with code 10
$ fwci run -w crystal-core -t boot --wait=30m

The limit is for the whole run, however many tests it has. With several DUTs, the run takes the first one that comes free.

If the agent can’t be reached, the run says so, naming the agent and where the DUT sets it. With other DUTs to choose from, it skips that one. Otherwise it runs none of the tests and exits with code 10. --wait only waits for a DUT someone holds, not for an agent it can’t reach.

While your run has the device, FirmwareCI treats it as a manual lock, and jobs for the device wait until your run frees it. A run waiting with --wait isn’t in FirmwareCI’s queue: it asks every 10 seconds, so it usually gets a freed device before a queued job does. A DUT with virtual or mock isn’t locked.

If the run can’t reach the agent to renew its lock for about 10 minutes, it stops the test before the lease runs out, so no one else gets the device while the test still uses it.

Stopping a Run

Ctrl-C stops the test and removes its container. If the test had started, fwci then runs the DUT’s post-stage, for at most 2 minutes, so the board doesn’t stay as the test left it:

Stopping: running the DUT's post-stage (Ctrl-C again skips it)…

Ctrl-C again skips the post-stage. Then fwci unlocks the device and exits with code 130. kill and closing the terminal do the same.

A run killed with kill -9 can’t clean up. Its container goes on running the test, and its files and the lock stay. A run that finds the agent gone as it unlocks the device leaves only the lock, and says so. Your next local run on this machine removes what is left and unlocks the device, and says so:

Freed Crystal-Core, which an earlier run left locked.

Until then, the lock ends with its lease, at most 15 minutes later, while a killed run’s container may still use the board. Remove that container with docker rm -f if you won’t run again soon.

Running in CI

In GitHub Actions and GitLab CI, the FirmwareCI action with local: true, or the GitLab CI template with FWCI_LOCAL: "true", sets this up for you: the action installs fwci, and both keep fwci.log and report.xml. See On the Runner. What follows holds for them too, and shows the steps to write yourself.

The exit code tells your CI how the run went:

Exit codeMeaning
0Every test passed.
1A test failed.
6A test couldn’t start: a broken test or DUT, a missing binary, storage, secret or key, an image the registry refused, no Docker.
10No device is available: someone else holds it, or its agent can’t be reached.
130The run was stopped.

If several apply, the first in this order wins: 130, 1, 6, 10. Wrong flags exit with 2.

Without a terminal, fwci redraws nothing: a step’s line appears when it ends, and a failed step’s whole log right after it. When nothing has printed for 30 seconds, such as during an image pull or a quiet step, it prints a line like Still running step "Flash" (2m): Erasing block 112, naming the step’s newest output line, so a CI log shows the run is alive. In GitHub Actions and GitLab CI, the output has colour, and each passed step’s line opens to its whole log. Elsewhere, FORCE_COLOR=1 turns colour on and NO_COLOR=1 off.

In GitHub Actions and GitLab CI, fwci writes the run’s log to fwci.log and a JUnit report to report.xml in the working directory, as the runner’s cache is gone when the job ends. --log <file> and --junit <file> name other files, and --log="" or --junit="" write none. In GitHub Actions, fwci also puts a table of the tests on the run’s summary page, with each failed step’s error and last lines, and an error annotation for each failed step on the job’s check. That takes no extra action and no permissions.

--json prints one JSON document for the whole run on stdout instead of the steps’ lines, with a report for every test, including the ones that didn’t run. The document holds every log, so fwci then writes no log file unless --log names one. See JSON Report.

A GitHub-hosted runner can’t reach a board in your lab. For a board, use a self-hosted runner that reaches its dutagent, or run the tests on FirmwareCI with --synced, see Running a Job. A step in a GitHub Actions workflow, on a runner with Docker and fwci:

- name: Boot test
  run: fwci run -w crystal-core --wait=30m Binary=build/coreboot.rom
- uses: actions/upload-artifact@v7
  if: always()
  with:
    name: fwci
    path: |
      fwci.log
      report.xml

Upload both files even when the job fails. In GitLab, the report appears in the pipeline’s Tests tab and in the merge request once the job declares it:

boot-test:
  script:
    - fwci run -w crystal-core --wait=30m Binary=build/coreboot.rom
  artifacts:
    when: always
    paths: [fwci.log]
    reports:
      junit: report.xml

--wait=30m waits for a board someone else holds instead of failing with exit code 10. Set the CI job’s own timeout above the wait and the tests’ timeouts.

Cancelling a CI job gives fwci only a few seconds: GitHub sends Ctrl-C, then stops the job 10 seconds later, and GitLab about as fast. That’s too short for the post-stage, so a board can be left as the test left it. The next local run on the same machine frees the lock, unless fwci runs in a container, as with a docker executor, or the runner’s cache is gone. Then the lock ends with its lease, at most 15 minutes later.

A local run needs Docker on the machine fwci runs on. GitLab’s Docker-in-Docker service runs Docker in another container, which a local run refuses. Use a shell executor, or a docker executor that gives the job the host’s Docker socket, network and cache directory, see In a Container.

See Also