Notifications

Overview

FirmwareCI can tell your team when something happens: a job fails, a rerun fixes a failure, a device goes offline, or a configuration push does not sync. It sends these notices to Slack, to an HTTPS endpoint of your own (a webhook), or by email.

Two things set this up:

  • A channel says where a notice goes: a Slack channel, a webhook URL or an email address.
  • A rule says which events go to which channels, and can limit them to some projects, workflows or branches.

Only organization admins manage channels and rules. Use the web UI under Settings → Notifications, or the CLI command fwci notification (see CLI).

Channels

Slack

  1. In Slack, add an incoming webhook for the channel that should get the notices, and copy its URL. It starts with https://hooks.slack.com/services/.
  2. In FirmwareCI, open Settings → Notifications. Under New channel, enter a name, set Kind to Slack, paste the URL and click Add channel.

FirmwareCI first sends a test message to Slack. It saves the channel only when that message arrives. The webhook URL is a secret: FirmwareCI stores it encrypted and shows only its end.

Webhook

A webhook channel posts each event as JSON to a URL of your own. Use it to connect FirmwareCI to your own services, for example to open a ticket when a job request fails.

The receiver must meet these conditions:

  • The URL starts with https:// and contains no user name or password.
  • The host resolves to public internet addresses only. FirmwareCI refuses private, loopback, link-local and tailnet addresses.
  • The receiver answers within 10 seconds with a status code from 200 to 299. FirmwareCI does not follow redirects.

To add it:

  1. Under New channel, enter a name, set Kind to Webhook, paste the URL and click Add channel. FirmwareCI first sends a notification.test event, and saves the channel only when it arrives.
  2. FirmwareCI shows the signing key of the channel, for example whsec_37d1…. Click it to copy it, and store it with your receiver. FirmwareCI shows it only once. If you lose it, delete the channel and add it again.

Slack, Microsoft Teams and Discord URLs do not work as webhooks: they expect their own message format. For Slack, use the Slack channel kind.

Requests

Each event is one POST request with a JSON body and these headers:

HeaderValue
Content-Typeapplication/json
User-AgentFirmwareCI-Webhook/1
X-FirmwareCI-EventThe event type, for example job.failed
X-FirmwareCI-DeliveryThe ID of the event. A retry sends the same ID, so use it to drop duplicates.
X-FirmwareCI-Signaturesha256= and the hex HMAC-SHA256 of the body, computed with the signing key

Check the signature

Anyone who knows the URL of your receiver can send it requests. The signature lets the receiver tell the requests of FirmwareCI from fakes. Compute the HMAC-SHA256 of the raw request body with the full signing key (including whsec_), and compare it with the header in constant time:

import hashlib
import hmac


def is_from_firmwareci(body: bytes, signature: str, key: str) -> bool:
    expected = "sha256=" + hmac.new(key.encode(), body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)

To check one saved request by hand:

openssl dgst -sha256 -hmac "$FWCI_SIGNING_KEY" body.json

Payload

{
  "id": "01M617TBTCED1S7FM900P164VZ",
  "type": "job.failed",
  "version": 1,
  "org_id": "01KVSXXRP06EAZKM211CQAVEBJ",
  "occurred_at": "2026-10-28T12:30:04.619474058Z",
  "subject": { "kind": "job", "id": "01M617T98A8TVS5GSDFSW634ZC" },
  "data": {
    "organization": "acme",
    "project_id": "01M3VAQ3NCYXTMEC2ZNV7AFPMX",
    "workflow": "boot",
    "test": "boot-linux",
    "device": "rack-7",
    "status": "failed"
  },
  "links": { "web": "https://app.firmware-ci.com/acme/jobs/01M617T98A8TVS5GSDFSW634ZC" }
}
FieldMeaning
idThe ID of the event, the same as X-FirmwareCI-Delivery
typeThe event type, see Events
versionThe version of this format, now 1
org_id, occurred_atThe organization, and when the event happened
subjectWhat the event is about: kind is job, job_request, device or project, and id is its ID
dataThe facts of the event. Only the fields that apply are present.
links.webThe page of the subject in FirmwareCI. For a failed config sync, the commit in your repository.

The fields of data:

FieldPresent for
organizationEvery event
project_idJob, job request and config sync events
workflow, statusJob and job request events
test, deviceJob events, and device for device events
branchJob and job request events of runs started from CI (the source branch), and config sync events
repository, commitConfig sync events: the pushed revision
errorFailed jobs, failed config syncs and device outages: what went wrong
summaryJob events that the failure classifier rated, such as an infrastructure failure
reason, by, untilDisabled devices: why, by whom, and until when

Retries

  • Retried: answers 408, 429 and 500 to 599, and network errors other than a refused address. FirmwareCI tries again after about 1, 5, 15 and 40 minutes, then gives up. On a 429 it waits at least as long as Retry-After says, up to one hour.
  • Not retried: all other answers from 300 to 499. The delivery fails at once.
  • Channel off: after 10 failed deliveries in a row, the channel turns itself off. Send test turns it on again when the test message arrives.

Email

  1. Under New channel, enter a name, set Kind to Email, enter one address (for example team@example.com) and click Add channel.
  2. FirmwareCI mails a confirmation link to that address. The link works for 7 days. The channel sends nothing until someone opens the link and confirms.

Each notice mail has an unsubscribe link. After an unsubscribe, the channel sends nothing to that address.

  • Send new link mails a new confirmation link, at most once every 5 minutes.
  • To an address that unsubscribed, it sends at most one link a day. The channel stays unsubscribed until the person confirms that link.
  • An organization can have only one email channel for each address.

Channel states

StateMeaning
OnThe channel gets notices.
Waiting for confirmationAn email channel whose address has not confirmed yet.
UnsubscribedThe address unsubscribed. The channel sends nothing.
N failed in a rowThe last deliveries failed. After 10, the channel turns off.
Off: reasonThe channel turned itself off. Send test turns it on again.

You cannot delete a channel while a rule sends to it. Change or delete the rule first. Deliveries to a deleted channel that are still pending are not sent.

Rules

A rule has three parts: When names the events, Send to the channels that get them, and Trigger the projects, workflows and branches they must come from. Each part is a table with one row per subject, such as Job or Slack.

To add a rule, click New rule under the rules, and fill in the form that opens:

  1. Enter a name.
  2. Under When, click the events you want. A picked event is filled dark. A click on a row name, such as Job, picks or drops all events of the row. Point at an event to see when it fires.
  3. Under Send to, click the channels.
  4. Under Trigger, pick projects, workflows and branches, or none (see Filters).
  5. Click Add rule.

On a rule card, Edit opens the same form. The switch turns a rule on or off. Mute turns it off for 1 hour, 8 hours, 1 day or 1 week; while it is muted, click the button again to unmute it.

Events

EventWhen it fires
Job succeededA single test passed.
Job failedA single test failed.
Job abortedA single test stopped before its end: someone aborted it, it timed out, or its device went offline.
Job fixedA test passed on a new attempt in its request after it had failed: Rerun job, or a rerun of the unsuccessful tests. You get this instead of Job succeeded.
Job infrastructure failureThe failure classifier blames a failed test on the lab, not on the firmware, for example the network. You get this after Job failed.
Job request succeededA workflow run passed all its tests.
Job request failedAt least one test of a workflow run failed. Only the newest attempt of each test counts.
Job request abortedA workflow run stopped: no test failed, but at least one was aborted.
Device offlineA device missed 3 health checks in a row, which takes about 3 minutes.
Device onlineAn offline device answers its health check again.
Device disabledSomeone took a device out of scheduling, with a reason and maybe an end date.
Device enabledSomeone put a disabled device back into scheduling.
Config sync failedA push changed the FirmwareCI configuration, but it did not validate or sync. The commit check shows why.

A job is one test on one device. A job request is one run of a workflow with all its tests. A fix is a success too: a rule with Job succeeded also gets Job fixed.

Only a new attempt in the same request can fix a test. A rerun of the whole request starts a new request, so its tests report Job succeeded, not Job fixed.

Filters

  • Project: only events of these projects.
  • Workflow: only runs of workflows with these names, in any project.
  • Branch: only runs whose source branch matches. * matches any text, so release-* matches release-1.2. Type a pattern into the branch row and press Enter.

A row with no pick matches everything. When several rows have picks, an event must match all of them.

Two cases get no notices from a rule with filters:

  • Runs without a branch: only runs started from CI know their source branch. Runs started with the CLI or in the web UI have none, so a rule with a branch filter never gets them.
  • Device events: they belong to no project, workflow or branch. A rule with any filter gets no device events, so use a rule without filters for them.

Limits

LimitValue
Channels per organization100
Rules per organization100
Events, channels, projects, workflows or branch patterns in one rule50 each
Length of a workflow name or branch pattern in a rule200 characters
Length of a channel or rule name100 characters

CLI

fwci notification (short: fwci notify) manages channels and rules from the command line. Every command takes -o json.

# The events a rule can take
fwci notification events

# Channels
fwci notification channel ls
fwci notification channel add team-slack --kind slack       # asks for the URL, which stays hidden
echo "$WEBHOOK_URL" | fwci notification channel add ci-hook --kind webhook
fwci notification channel add team-mail --kind email --email team@example.com
fwci notification channel test <channel-id>                 # also turns an off channel on again
fwci notification channel resend <channel-id>               # a new confirmation link
fwci notification channel rm <channel-id>

# Rules
fwci notification rule ls
fwci notification rule add release-failures \
  --event job.failed --event job.fixed \
  --channel <channel-id> --branch 'release-*'
fwci notification rule mute <rule-id> --for 2h              # --for 0 unmutes
fwci notification rule disable <rule-id>
fwci notification rule enable <rule-id>
fwci notification rule rm <rule-id>

The Slack and webhook URLs are secrets. The CLI reads them from a hidden prompt or from standard input, never from a flag, so they stay out of your shell history.