Local Runs

fwci run runs a workflow’s tests on your own machine, without FirmwareCI. This page lists what a local run takes and what it does. For a walk-through, see Running Tests Locally. With --remote or --synced, fwci run runs the tests on FirmwareCI instead, see Running a Job.

fwci run -w <workflow> [-t <test>]... [flags] [<key>=<file>]...

Run it inside the repository whose .firmwareci holds the workflow. The tests are the workflow’s: the files under its tests directory, picked by the name in each. A test file that doesn’t validate stops the run. Before the first test, fwci checks every test, and if one can’t start, none runs.

Flags

FlagDescription
-w <workflow>The workflow, by its directory under .firmwareci/workflows or by the name in its workflow.yaml. Without it, fwci takes the only workflow, or asks on a terminal. If no workflow matches, fwci lists them.
-t <test>A test, by the name in its file. Repeat it, or list names as -t flash,boot. The tests run in that order, and a name given twice runs twice. Without -t, every enabled test runs, in the order of their names. If no test matches, fwci lists the workflow’s tests.
<key>=<file>The file for [[input.<key>]]. Give one for each binary.
--dut <name>The DUT, by its directory under .firmwareci/duts or by the name in its dut.yaml. A path ending in duts/<dir> works too. Its label has to be the workflow’s runs-on. Without --dut, the run takes the first free DUT with that label, in the order of their names. If no DUT matches, fwci lists them.
--storage <name>=<directory>The files for the storage <name>, used as [[storage.<name>]]. Repeat for more.
--ssh-key <name>=<file>The private key for [[ssh-keys.<name>]]. Its public half is <file>.pub, if that exists. Repeat for more.
--waitIf someone else holds the device, wait until it’s free. With several DUTs, wait for the first that comes free.
--wait=<duration>Wait at most this long, for the whole run, such as --wait=30m or --wait 1h30m.
--verbose, -vShow every step’s whole log as it comes. Without it, a passed step is one line, and only a failed one shows its log.
--log <file>Write every step’s whole log to this file. Without it, the run writes one under ~/.cache/fwci/logs/, see Files, or fwci.log in GitHub Actions and GitLab CI, but none with --json, whose report holds every log. --log="" writes none.
--junit <file>Write a JUnit report when the run ends: a suite for each test, a case for each step. In GitHub Actions and GitLab CI it defaults to report.xml. --junit="" writes none.
--jsonPrint one JSON document for the run instead of the live output, see JSON Report.
--debugShow the engine’s warnings too, and its log lines in full, with times and fields.

-b, -p, --priority, --no-follow, --project-link, --commit, --action and --github-installation are for a run on FirmwareCI, so a run on this machine refuses them. --dut, --storage, --ssh-key and --wait are only for a run on this machine, so --remote and --synced refuse them.

Environment Variables

VariableDescription
FWCI_SECRET_<name>The value of [[secrets.<name>]]. <name> matches the secret exactly, case included. An empty variable counts as not set.
FWCI_RUNNERA Linux x64 fwci of the same version to run inside the container, see The Runner.
DOCKER_HOST, DOCKER_CONTEXTChoose the Docker daemon as docker does: DOCKER_HOST, then DOCKER_CONTEXT, then the current context. The daemon has to run on this machine.
XDG_CACHE_HOMEWhere the run keeps its files, see Files. Defaults to ~/.cache.
GITHUB_ACTIONS, GITLAB_CISet by GitHub Actions and GitLab CI. fwci then writes fwci.log (not with --json) and report.xml in the working directory unless --log or --junit say otherwise, and folds each passed step’s log. In GitHub Actions it also adds the tests to the run’s summary page, through the file GITHUB_STEP_SUMMARY names, and an error annotation for each failed step.
NO_COLOR, FORCE_COLOR, CLICOLOR_FORCENO_COLOR turns colour off. FORCE_COLOR or CLICOLOR_FORCE turn it on without a terminal, as GitHub Actions and GitLab CI do on their own.

Exit Codes

CodeMeaning
0Every test passed.
1A test failed.
2Wrong flags, such as -b, which is only for a run on FirmwareCI.
6A test couldn’t start: a workflow, test or DUT that isn’t there or doesn’t validate, a DUT without the workflow’s label or none with it, a missing binary, storage, secret or SSH key, an image the registry refused or doesn’t have, no Docker, an unusable FWCI_RUNNER.
10No device is available: someone else holds each one the run may take, also after --wait=<duration>, or their agents can’t be reached.
130Stopped by Ctrl-C, kill, or a closed terminal.

If several apply to a run, the first in this order wins: 130, 1, 6, 10.

JSON Report

--json prints one document when the run ends, with an entry for every test, in the order they ran:

{
  "jobs": [
    {
      "test_name": "Boot",
      "test_file": ".firmwareci/workflows/workflow-crystal-core/tests/boot.yaml",
      "dut": "dut-crystal-core",
      "status": "failed",
      "duration": "2m14s",
      "stages": [ ... ],
      "result": {}
    }
  ]
}
FieldDescription
test_nameThe test’s name.
test_fileThe test’s file.
dutThe DUT’s directory under .firmwareci/duts.
statussucceeded, failed, not_started (the run couldn’t start it), unavailable (the device wasn’t available) or stopped.
stagesEvery stage and step with its log, as fwci view --json --verbose prints them.
errorWhy the test didn’t finish, if it didn’t.

The human output goes to stderr, so stdout holds only the document. When the run can’t name its tests, such as for a workflow or test that isn’t there, jobs is empty and error says why.

Templates

TemplateIn a local run
[[input.<key>]]The file from <key>=<file>.
[[attributes.<name>]]The DUT’s dut.yaml, as in a job.
[[reservation.<name>]]The reservation system, once the device is locked.
[[storage.<name>]]A copy of the directory from --storage <name>=<directory>. commands and paths from storage.yaml apply.
[[secrets.<name>]]FWCI_SECRET_<name>. A secret that isn’t set stops the run before any test runs.
[[ssh-keys.<name>]]The key from --ssh-key <name>=<file>. A test that names no key tries every key passed and the default key.
[[ssh-keys.default]]The key pair built into fwci, the one the Test Images accept. The container also has it at /root/.ssh/fwci, as in a job. --ssh-key default=<file> replaces it, for an instance with its own default key.

See Templating & Variables for the syntax.

Job Image

The tests run in the image of the workflow’s workflow.yaml. Without one, they run in the default job image. The default image is public, so it needs no login. See Job Images.

For a private image, the credentials come from, in this order:

  1. The workflow’s username and password. A [[secrets.<name>]] in them reads FWCI_SECRET_<name>.
  2. Your docker login for the registry.

If the registry doesn’t confirm the image, the run uses a copy on your machine if there is one, and otherwise fails, naming what to fix. A job on the server always asks the registry. fwci doesn’t send identity tokens, such as the ones az acr login stores. For such a registry, docker pull the image before the run.

Reservation

reservation-systemIn a local run
dutctlThe run locks the device on its agent as <user>@<host>/<run-ID> before the first test and unlocks it after the last one. The lease is 15 minutes, renewed every 5 minutes until the run ends, also between tests. If two renewals in a row fail while a test runs, the run stops the test before the lease runs out.
virtualNothing is locked, and local runs don’t wait for each other.
mockNothing is locked.

The agent has to be reachable from your machine. A FirmwareCI job sees the lock of a local run as a manual lock and waits.

If someone else holds the device, the run exits with code 10 before any test runs. With several DUTs to choose from, it tries them in turn, skips one whose agent it can’t reach, and exits with code 10 only when someone holds each of the others. The message says who the holder is, and with one DUT when the holder’s lease ends at the latest:

The message saysThe lock’s owner
FirmwareCI job <job-id><job-id>@firmwareci
another run of yours (PID <pid>)<user>@<host>/<run-ID> of your user on this machine
a local run of <user>@<host>, <run-ID>a local run of another user or machine
the owner as it isa lock taken by hand

With --wait, the run asks for the device again every 10 seconds, and prints a line whenever a holder changes. Ctrl-C stops the wait. A waiting run isn’t in FirmwareCI’s queue, so it usually gets a freed device before a queued job does, which only notices on its next check, about once a minute.

Files

PathHolds
~/.cache/fwci/jobs/<job-ID>/A test’s storage copies and SSH keys. Removed when the test ends.
~/.cache/fwci/locks/<run-ID>The device a run holds. Removed once the device is unlocked.
~/.cache/fwci/logs/<date-time>-<dut>-….logEvery step’s whole log of a run, a test after the other, without colours. Each test’s summary shows the path. The newest 50 are kept. --log writes it elsewhere, and a run with --json writes none.
fwci.log, report.xml in the working directoryIn GitHub Actions and GitLab CI: the run’s log and its JUnit report, see Flags.

The container runs as root. If a test writes into a storage copy, fwci can’t remove those files, and prints where they are, to remove them with sudo rm -rf.

A run killed with kill -9 leaves its container, files and lock behind, and the container goes on running the test. A run that can’t reach the agent to unlock the device leaves only the lock. Your next local run on this machine removes what is left and unlocks the device, if the run that left it isn’t running anymore. Runs of a fwci in another container are left alone, as fwci can’t tell if they are still running.

Ctrl-C, kill and a closed terminal stop the test at once and remove its container. If the test had started, fwci then runs the post-stage as a job of its own, under the run’s lock, for at most 2 minutes. A second stop skips it. Then fwci unlocks the device and exits with code 130.

Network

The container uses the host’s network, so a test reaches what your machine reaches, such as a VPN and the names only its DNS resolves. The runner inside the container listens on a free port on 127.0.0.1.

A port that a test opens is open on your machine too. A QEMU hostfwd=tcp::2222-:22 listens on all of your machine’s interfaces. Write hostfwd=tcp:127.0.0.1:2222-:22 to keep it on your machine. Two runs that forward the same port can’t run at the same time.

A DOCKER_HOST or Docker context that points at another machine is refused, as the runner’s API would be open on that machine.

The Runner

fwci copies itself into the container and runs the test from there. That takes a static Linux x64 build, as the x64 release is. On Linux ARM64, fwci refuses a local run until FWCI_RUNNER names the fwci of the same version from the Linux x64 .tar.gz on the Downloads page. fwci only checks that it is a Linux x64 program, not its version.

$ tar -xzf fwci-latest.linux-amd64.tar.gz -C $HOME
$ export FWCI_RUNNER=$HOME/fwci-latest.linux-amd64

Job images are Linux x64 too, so Docker on such a machine needs x64 emulation.

In a Container

When fwci runs in a container, as in some CI setups, it starts the test’s container through the same Docker daemon. Start the container that runs fwci with:

  • the Docker socket mounted,
  • --network host, so the runner’s port is reachable,
  • the cache directory mounted at the same path inside as outside, as the daemon mounts the run’s files by their path on the host.

Compared to a Job on the Server

Job on the serverLocal run
ResultsDashboard, commit status, notificationsYour terminal or --json only
LoginRequiredNot needed
Storage, secrets, SSH keysFrom FirmwareCIFrom your machine, see Templates
NetworkThe job’s ownYour machine’s
Image the registry doesn’t confirmThe job failsA copy on your machine is used
Device lock<job-id>@firmwareci, 8-hour lease<user>@<host>/<run-ID>, 15-minute lease renewed while the run lasts

dut.yaml, pre.yaml and post.yaml are checked as a push checks them, so a DUT that runs here isn’t refused when you push it.