Notifications
Overview
FirmwareCI can tell your team when something happens: a job fails, a rerun fixes a failure, a device goes offline, or a configuration push does not sync. It sends these notices to Slack, to an HTTPS endpoint of your own (a webhook), or by email.
Two things set this up:
- A channel says where a notice goes: a Slack channel, a webhook URL or an email address.
- A rule says which events go to which channels, and can limit them to some projects, workflows or branches.
Only organization admins manage channels and rules. Use the web UI under Settings → Notifications, or the CLI command fwci notification (see CLI).
Channels
Slack
- In Slack, add an incoming webhook for the channel that should get the notices, and copy its URL. It starts with
https://hooks.slack.com/services/. - In FirmwareCI, open Settings → Notifications. Under New channel, enter a name, set Kind to Slack, paste the URL and click Add channel.
FirmwareCI first sends a test message to Slack. It saves the channel only when that message arrives. The webhook URL is a secret: FirmwareCI stores it encrypted and shows only its end.
Webhook
A webhook channel posts each event as JSON to a URL of your own. Use it to connect FirmwareCI to your own services, for example to open a ticket when a job request fails.
The receiver must meet these conditions:
- The URL starts with
https://and contains no user name or password. - The host resolves to public internet addresses only. FirmwareCI refuses private, loopback, link-local and tailnet addresses.
- The receiver answers within 10 seconds with a status code from 200 to 299. FirmwareCI does not follow redirects.
To add it:
- Under New channel, enter a name, set Kind to Webhook, paste the URL and click Add channel. FirmwareCI first sends a
notification.testevent, and saves the channel only when it arrives. - FirmwareCI shows the signing key of the channel, for example
whsec_37d1…. Click it to copy it, and store it with your receiver. FirmwareCI shows it only once. If you lose it, delete the channel and add it again.
Slack, Microsoft Teams and Discord URLs do not work as webhooks: they expect their own message format. For Slack, use the Slack channel kind.
Requests
Each event is one POST request with a JSON body and these headers:
| Header | Value |
|---|---|
Content-Type | application/json |
User-Agent | FirmwareCI-Webhook/1 |
X-FirmwareCI-Event | The event type, for example job.failed |
X-FirmwareCI-Delivery | The ID of the event. A retry sends the same ID, so use it to drop duplicates. |
X-FirmwareCI-Signature | sha256= and the hex HMAC-SHA256 of the body, computed with the signing key |
Check the signature
Anyone who knows the URL of your receiver can send it requests. The signature lets the receiver tell the requests of FirmwareCI from fakes. Compute the HMAC-SHA256 of the raw request body with the full signing key (including whsec_), and compare it with the header in constant time:
To check one saved request by hand:
Payload
| Field | Meaning |
|---|---|
id | The ID of the event, the same as X-FirmwareCI-Delivery |
type | The event type, see Events |
version | The version of this format, now 1 |
org_id, occurred_at | The organization, and when the event happened |
subject | What the event is about: kind is job, job_request, device or project, and id is its ID |
data | The facts of the event. Only the fields that apply are present. |
links.web | The page of the subject in FirmwareCI. For a failed config sync, the commit in your repository. |
The fields of data:
| Field | Present for |
|---|---|
organization | Every event |
project_id | Job, job request and config sync events |
workflow, status | Job and job request events |
test, device | Job events, and device for device events |
branch | Job and job request events of runs started from CI (the source branch), and config sync events |
repository, commit | Config sync events: the pushed revision |
error | Failed jobs, failed config syncs and device outages: what went wrong |
summary | Job events that the failure classifier rated, such as an infrastructure failure |
reason, by, until | Disabled devices: why, by whom, and until when |
Retries
- Retried: answers 408, 429 and 500 to 599, and network errors other than a refused address. FirmwareCI tries again after about 1, 5, 15 and 40 minutes, then gives up. On a 429 it waits at least as long as
Retry-Aftersays, up to one hour. - Not retried: all other answers from 300 to 499. The delivery fails at once.
- Channel off: after 10 failed deliveries in a row, the channel turns itself off. Send test turns it on again when the test message arrives.
- Under New channel, enter a name, set Kind to Email, enter one address (for example
team@example.com) and click Add channel. - FirmwareCI mails a confirmation link to that address. The link works for 7 days. The channel sends nothing until someone opens the link and confirms.
Each notice mail has an unsubscribe link. After an unsubscribe, the channel sends nothing to that address.
- Send new link mails a new confirmation link, at most once every 5 minutes.
- To an address that unsubscribed, it sends at most one link a day. The channel stays unsubscribed until the person confirms that link.
- An organization can have only one email channel for each address.
Channel states
| State | Meaning |
|---|---|
| On | The channel gets notices. |
| Waiting for confirmation | An email channel whose address has not confirmed yet. |
| Unsubscribed | The address unsubscribed. The channel sends nothing. |
| N failed in a row | The last deliveries failed. After 10, the channel turns off. |
| Off: reason | The channel turned itself off. Send test turns it on again. |
You cannot delete a channel while a rule sends to it. Change or delete the rule first. Deliveries to a deleted channel that are still pending are not sent.
Rules
A rule has three parts: When names the events, Send to the channels that get them, and Trigger the projects, workflows and branches they must come from. Each part is a table with one row per subject, such as Job or Slack.
To add a rule, click New rule under the rules, and fill in the form that opens:
- Enter a name.
- Under When, click the events you want. A picked event is filled dark. A click on a row name, such as Job, picks or drops all events of the row. Point at an event to see when it fires.
- Under Send to, click the channels.
- Under Trigger, pick projects, workflows and branches, or none (see Filters).
- Click Add rule.
On a rule card, Edit opens the same form. The switch turns a rule on or off. Mute turns it off for 1 hour, 8 hours, 1 day or 1 week; while it is muted, click the button again to unmute it.
Events
| Event | When it fires |
|---|---|
| Job succeeded | A single test passed. |
| Job failed | A single test failed. |
| Job aborted | A single test stopped before its end: someone aborted it, it timed out, or its device went offline. |
| Job fixed | A test passed on a new attempt in its request after it had failed: Rerun job, or a rerun of the unsuccessful tests. You get this instead of Job succeeded. |
| Job infrastructure failure | The failure classifier blames a failed test on the lab, not on the firmware, for example the network. You get this after Job failed. |
| Job request succeeded | A workflow run passed all its tests. |
| Job request failed | At least one test of a workflow run failed. Only the newest attempt of each test counts. |
| Job request aborted | A workflow run stopped: no test failed, but at least one was aborted. |
| Device offline | A device missed 3 health checks in a row, which takes about 3 minutes. |
| Device online | An offline device answers its health check again. |
| Device disabled | Someone took a device out of scheduling, with a reason and maybe an end date. |
| Device enabled | Someone put a disabled device back into scheduling. |
| Config sync failed | A push changed the FirmwareCI configuration, but it did not validate or sync. The commit check shows why. |
A job is one test on one device. A job request is one run of a workflow with all its tests. A fix is a success too: a rule with Job succeeded also gets Job fixed.
Only a new attempt in the same request can fix a test. A rerun of the whole request starts a new request, so its tests report Job succeeded, not Job fixed.
Filters
- Project: only events of these projects.
- Workflow: only runs of workflows with these names, in any project.
- Branch: only runs whose source branch matches.
*matches any text, sorelease-*matchesrelease-1.2. Type a pattern into the branch row and press Enter.
A row with no pick matches everything. When several rows have picks, an event must match all of them.
Two cases get no notices from a rule with filters:
- Runs without a branch: only runs started from CI know their source branch. Runs started with the CLI or in the web UI have none, so a rule with a branch filter never gets them.
- Device events: they belong to no project, workflow or branch. A rule with any filter gets no device events, so use a rule without filters for them.
Limits
| Limit | Value |
|---|---|
| Channels per organization | 100 |
| Rules per organization | 100 |
| Events, channels, projects, workflows or branch patterns in one rule | 50 each |
| Length of a workflow name or branch pattern in a rule | 200 characters |
| Length of a channel or rule name | 100 characters |
CLI
fwci notification (short: fwci notify) manages channels and rules from the command line. Every command takes -o json.
The Slack and webhook URLs are secrets. The CLI reads them from a hidden prompt or from standard input, never from a flag, so they stay out of your shell history.