Overview

What is FirmwareCI?

FirmwareCI is an automated test system for firmware. Firmware is software that runs on electronic devices and manages their fundamental functions. On PCs and servers, firmware is commonly known as BIOS.

Testing firmware requires two steps: first, transfer the firmware to the device’s flash memory using specialized equipment; second, verify that all device functions work correctly.

Most testing approaches in firmware development rely on source code analysis. FirmwareCI takes a different approach by enabling direct hardware testing, treating the firmware as a black box.

Key Concepts

Black-Box Testing

FirmwareCI tests compiled firmware binaries on actual hardware without requiring access to source code. This approach validates the final product that users will run, catching integration issues that unit tests might miss.

Automated Hardware Testing

Tests run automatically on every commit, pull request, and release. FirmwareCI manages hardware reservations, test execution, and reporting without manual intervention.

Test Steps

Tests are defined using predefined Test Step Commands - reusable building blocks that handle complex operations. Commands range from simple bash execution to firmware scanning tools like ChipSec and Binarly.

YAML Configuration

All configuration uses straightforward YAML files:

  • Test files: Define stages and steps for validation scenarios
  • Workflow configuration: Specifies which tests run on which hardware
  • DUT files: Configure devices under test (DUTs) and their interfaces

Device Under Test (DUT)

A DUT is hardware connected through control interfaces that enable:

  • Power cycling
  • Serial console access
  • Flash chip read/write operations
  • Binary execution

DUTs are managed by the FirmwareCI server and allocated to test runs automatically.

How FirmwareCI Works

Workflow Execution

  1. Trigger: A commit, pull request, or manual request triggers a workflow
  2. Reservation: FirmwareCI reserves available DUTs matching the workflow requirements
  3. Pre-stage: Hardware is prepared (flash firmware, configure settings)
  4. Test execution: Test stages run sequentially, with steps executing within each stage
  5. Post-stage: Hardware is cleaned up and returned to the pool
  6. Reporting: Test results are collected and reported back to your CI system

Test Structure

Tests consist of stages containing steps:

stages:
  - name: Boot Test
    steps:
      - cmd: ping
        name: Wait for device
        parameters:
          host: "[[attributes.Host]]"

      - cmd: cmd
        name: Check OS version
        transport: *ssh_transport
        parameters:
          executable: cat
          args: ["/etc/os-release"]

Each step must succeed for the stage to complete. If any step fails, the stage fails and test execution moves to the next stage.

Benefits

Scalability: Run tests on multiple devices simultaneously, scaling as your project grows.

Integration: Works with GitHub Actions, GitLab CI, and direct API calls with minimal configuration.

Reusability: Predefined test steps reduce boilerplate and speed up test creation.

Standardization: Clear structure for firmware testing promotes best practices across teams.

Real Hardware: Tests run on actual hardware, catching issues that emulation might miss.

Next Steps