GitHub Actions and GitLab CI

Your pipeline runs a workflow’s tests with the FirmwareCI action in GitHub Actions, or with a template job in GitLab CI. Both run fwci run and take the same switches. By default the tests run on FirmwareCI, on one of your organization’s DUTs with the workflow’s label, and FirmwareCI reports them on the commit. With local: true they run on the runner itself, next to your board.

Before the first run on FirmwareCI, set up:

GitHub Actions

- uses: actions/checkout@v7
- run: make build
- uses: BlindspotSoftware/actions@v8
  with:
    workflow: crystal-core
    binaries: Binary=build/coreboot.rom
    token: ${{ secrets.FWCI_TOKEN }}

This runs every test of the workflow crystal-core, as FirmwareCI synced it from the commit the run is on, with build/coreboot.rom for [[input.Binary]]. The step ends once the tests are queued, and the FirmwareCI GitHub App reports their results on the commit.

You wantAdd
Some tests, not alltests: boot, flash, by the name in their files
The step to wait for the resultsfollow: true
This checkout’s tests on FirmwareCI, not the synced onessynced: false
The run on this runner, not FirmwareCIlocal: true
A particular DUT for a run on this runnerdut: board-2; without it, the first free DUT with the workflow’s label

The workflow is the name in its workflow.yaml. The Workflow filter on the Job Requests page of the web UI lists the names too.

On FirmwareCI

FirmwareCI finds the workflow in this repository’s project, on the branch or tag the run is on, and runs each test as a job on any free DUT with the workflow’s label. The FirmwareCI app reports each test as a check on the commit, which links the job and the run’s JUnit report. With follow: true the step waits for the jobs: it fails on a failed test, the job’s own check gets an annotation for each failed step, the run’s summary page a table of the tests, and fwci.log and report.xml are kept as an artifact. The runner is held while the jobs queue. Each job and matrix leg keeps its own artifact; when the action runs twice in one job, set artifact, such as artifact: fwci-${{ matrix.board }}.

synced: false sends this checkout’s test files instead, as a one-shot run that no commit reports. The workflow, its label and its DUTs still come from FirmwareCI, so the workflow has to be pushed once.

FirmwareCI picks the DUT, so dut is refused there: give a board a label of its own to run on it alone.

On pull_request events, leave commit_hash unset: the action reports on the pull request’s head. ${{ github.sha }} is GitHub’s merge commit, not the commit under test.

On the Runner

With local: true the tests come from this checkout’s .firmwareci and run one after another on one DUT, as in Running Tests Locally, with no FirmwareCI login. The runner needs Docker and has to reach the board: a GitHub-hosted runner reaches only virtual and mock DUTs, so a board needs a self-hosted runner next to it. Results show as with follow: true, and the artifact is kept also when a test fails. A test’s [[secrets.X]] comes from FWCI_SECRET_X in the step’s env.

- uses: BlindspotSoftware/actions@v8
  with:
    workflow: crystal-core
    binaries: Binary=build/coreboot.rom
    local: true
    wait: 30m

In a public repository, a pull request from a fork brings its own .firmwareci, and its tests would run on your board with the runner’s Docker. Run such a job only on push, or require approval for workflows from outside contributors. Artifacts of a public repository are public too.

The step’s exit code says how it went: 0 every test passed, 1 a test failed, 6 a test couldn’t start, 10 every DUT is held by someone else or its agent can’t be reached. wait waits only for held DUTs.

Inputs

InputDescription
workflowThe workflow to run, by name. Required.
testsTest names, separated by commas or newlines. Default: all of the workflow’s
binariesname=file, or name=URL on FirmwareCI, separated by ; or newlines, for [[input.name]].
localtrue runs the tests on this runner instead of FirmwareCI. Default: false
syncedOn FirmwareCI, the tests as FirmwareCI synced them, reported on the commit. false sends this checkout’s tests, as a one-shot run not reported on a commit. Default: true
argsMore arguments for fwci run, such as --verbose.
working_directoryWhere fwci runs, and binaries are relative to. With local: true or synced: false, inside the repository whose .firmwareci holds the workflow. Default: .
artifactName of the artifact with fwci.log and report.xml, with local: true or follow. Empty keeps none. Default: fwci-<job>-<matrix index>
versionThe fwci to install, 0.21.0 or newer. installed uses the one on the PATH, as on a self-hosted runner that has it. Default: latest
On FirmwareCI
tokenA FirmwareCI organization or user API token.
email, passwordLog in with these instead of a token.
organizationThe organization to submit to, when the credential reaches more than one. An organization token names its own.
followtrue waits for the jobs, see above. Default: false
prioritylow, mid or high. Unset, FirmwareCI queues the run at mid, or at high with synced: false, see Queue Priority.
branchThe branch to find the workflow on. Defaults to the branch or tag the run is on.
project_linkThe repository holding the FirmwareCI config, when it isn’t this one, see Config in Another Repository. Needs synced (the default).
commit_hashThe commit to report on. Defaults to the pull request’s head, or the commit the run is on. Needs synced.
actionA label for the run, such as nightly or release. Needs synced.
github_installation_idThe FirmwareCI GitHub App’s installation, when project_link is in another GitHub organization. Needs synced.
apiThe FirmwareCI API, for a self-hosted instance. Default: https://api.firmware-ci.com
On the runner
dutThe DUT, by its directory under .firmwareci/duts or its name. It must carry the workflow’s label.
waittrue waits while someone else holds the DUTs, 30m at most that long.
storagename=directory, separated by ; or newlines, for [[storage.name]].
ssh_keysname=private key file, separated by ; or newlines, for [[ssh-keys.name]].

local, synced and follow take true or false, and empty keeps the default. Anything else stops the step, and so does local: true with synced: false.

The action installs fwci on Linux x64 runners. On another runner, install fwci yourself, see Installation, and set version: installed.

Authentication

In the web UI, Profile in the account menu opens your settings. Create a token under API Tokens there. An organization admin can create an organization token under the organization’s API Tokens on the same page. Add the token as the secret FWCI_TOKEN (Settings, Secrets and variables, Actions). Pass it as token. The action hands it to fwci for that step alone, so no login stays on the runner. email and password work too. A run on the runner needs none.

An organization token is the better CI credential: it names its organization and can’t reach another. A user token, or email and password, for an account in several organizations needs organization, by the name fwci org list shows.

GitLab CI

A template has a job to extend, with the same switches as variables:

include:
  - remote: https://raw.githubusercontent.com/BlindspotSoftware/actions/v8/templates/fwci.gitlab-ci.yml

firmware-test:
  extends: .fwci
  variables:
    FWCI_WORKFLOW: crystal-core
    FWCI_BINARIES: Binary=build/coreboot.rom

The job runs the firmwareci/action:v8 image, which follows the newest fwci within a day; firmwareci/action:0.21.0 and so on keep one. Set FWCI_TOKEN as a masked CI/CD variable.

VariableDescription
FWCI_WORKFLOWThe workflow to run, by name. Required.
FWCI_TESTSTest names, separated by commas or newlines. Default: all of the workflow’s
FWCI_BINARIESname=file, or name=URL on FirmwareCI, separated by ; or newlines.
FWCI_LOCALtrue runs the tests on this runner instead of FirmwareCI. Default: false
FWCI_SYNCEDOn FirmwareCI, the tests as FirmwareCI synced them, reported on the commit. false sends this checkout’s tests. Default: true
FWCI_FOLLOWtrue waits for the jobs and fails on a failed test. Default: false
FWCI_DUTWith FWCI_LOCAL: "true", the DUT. Default: the first free one with the workflow’s label
FWCI_ARGSMore fwci run flags, such as --priority low, -b main, --wait=30m or --storage OS=images.
FWCI_TOKEN, FWCI_EMAIL, FWCI_PASSWORDThe login, as for the action.
FWCI_ORGThe organization, when the credential reaches more than one.
FWCI_APIThe FirmwareCI API, for a self-hosted instance. Default: https://api.firmware-ci.com
FWCI_GITLAB_TOKENOnly if the project’s GitLab integration has no token: a project access token with api scope, for FirmwareCI to report back.

FWCI_LOCAL, FWCI_SYNCED and FWCI_FOLLOW take true, 1 or yes, or false, 0 or no, and empty keeps the default. Anything else stops the job, and so does FWCI_LOCAL true with FWCI_SYNCED false. Quote the value, as in FWCI_LOCAL: "true", since GitLab refuses an unquoted true or false. When the job waits for the tests, GitLab shows report.xml in the merge request. Otherwise fwci writes neither fwci.log nor report.xml, so GitLab’s warning that it found no files to upload is expected.

To run the tests on a runner next to your board, pick that runner with tags:

board-test:
  extends: .fwci
  tags: [lab]
  variables:
    FWCI_WORKFLOW: crystal-core
    FWCI_BINARIES: Binary=build/coreboot.rom
    FWCI_LOCAL: "true"
    FWCI_ARGS: --wait=30m

A test’s [[secrets.X]] comes from the CI/CD variable FWCI_SECRET_X.

A run on the runner with a docker executor needs the host’s network and Docker socket, and a cache directory at the same path in the job as on the host, as fwci hands paths in it to the host’s Docker. In the runner’s config.toml:

[[runners]]
  executor = "docker"
  environment = ["XDG_CACHE_HOME=/var/cache/fwci"]
  [runners.docker]
    network_mode = "host"
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/var/cache/fwci:/var/cache/fwci"]

A shell executor with fwci and Docker installed needs nothing more.

Config in Another Repository

FirmwareCI looks for the workflow in the project of the repository the pipeline runs in. A common setup keeps the FirmwareCI config (.firmwareci/) in a dedicated repository, such as my-org/firmware-tests, and runs the pipeline in a product repository, such as my-org/my-product. Name the config’s repository with project_link, or --project-link in FWCI_ARGS. A job request’s page in the web UI shows the value under its title, such as github.com/my-org/firmware-tests.

ValueConfig on
github.com/org/repoGitHub, from any CI
gitlab.com/group/nameGitLab.com, from any CI
gitlab.example.com/group/name, or with https://A self-hosted GitLab
org/repoGitHub in GitHub Actions, GitLab.com in GitLab CI

project_link only decides where the workflow comes from. The results still go back to the commit the pipeline runs on. The credentials need both repositories:

  • GitHub: the FirmwareCI GitHub App must be installed in the organization that owns the config’s repository. When that is another organization than the pipeline’s, set github_installation_id to that installation.
  • GitLab: the project’s GitLab integration token, or FWCI_GITLAB_TOKEN, needs at least read_repository on the config’s repository, on the same GitLab instance.
- uses: BlindspotSoftware/actions@v8
  with:
    workflow: coreboot-tests
    project_link: github.com/my-org/firmware-tests
    binaries: Binary=build/coreboot.rom
    token: ${{ secrets.FWCI_TOKEN }}

Without the Action

In GitHub Actions and GitLab CI, a step that runs fwci run --synced does the same: it logs in with FWCI_TOKEN for that one command and reports on the commit, see Running a Job. In another CI it runs the tests as synced too, but FirmwareCI reports them on no commit.

Moving from v7

docker://firmwareci/action:v7.0 keeps working as before, with its upper-case inputs, and gets no more changes. To move to v8, use BlindspotSoftware/actions@v8 in GitHub Actions; the v8 image fails a docker:// step. In GitLab CI, extend .fwci instead of running /bin/action in the v7.0 image.

v7v8 inputv8 GitLab variable
WORKFLOW_NAMEworkflowFWCI_WORKFLOW
INDIVIDUAL_TESTStestsFWCI_TESTS
BINARIESbinariesFWCI_BINARIES
TOKEN, EMAIL, PASSWORDtoken, email, passwordFWCI_TOKEN, FWCI_EMAIL, FWCI_PASSWORD
ORGANIZATIONorganizationFWCI_ORG
APIapiFWCI_API
GITLAB_TOKENFWCI_GITLAB_TOKEN
PRIORITYpriority--priority in FWCI_ARGS
BRANCHbranch-b in FWCI_ARGS
PROJECT_LINKproject_link--project-link in FWCI_ARGS
COMMIT_HASHcommit_hash--commit in FWCI_ARGS
ACTIONaction--action in FWCI_ARGS
GITHUB_INSTALLATION_IDgithub_installation_id
CIgone, fwci tells the CI itselfgone