Test Execution Model

Overview

FirmwareCI test files are structured into stages containing steps. This hierarchical structure enables organized, sequential execution with clear success criteria and failure handling.

Test Structure

Hierarchy

Test File
├── Pre-Stage (optional)
├── Stage 1
│   ├── Step 1
│   ├── Step 2
│   └── Step 3
├── Stage 2
│   ├── Step 1
│   └── Step 2
└── Post-Stage (optional)

Stages

Stages are named groups of related operations executed sequentially. Each stage contains one or more steps that accomplish a specific testing objective.

Stages and Steps Stages and Steps

Example stages:

  • Boot Test: Verify device boots and network connectivity
  • Firmware Validation: Run security scans and integrity checks
  • Performance Benchmarking: Execute performance tests and collect metrics
  • Stress Testing: Apply load and verify stability

Stages execute in definition order. If a step fails within a stage, the stage fails and execution moves to the next stage.

Steps

Steps are individual test operations executed within a stage. FirmwareCI offers a comprehensive set of Test Step Commands ranging from simple bash commands to complex integrations like the Robot Test Framework.

Common step types:

  • Command execution: Run shell commands on the DUT
  • File operations: Copy files, create archives
  • Network operations: Ping devices, check connectivity
  • Firmware analysis: Scan with ChipSec, Binarly, FWHunt
  • Hardware control: Power cycle, flash firmware, access serial
  • Benchmarking: Run CPU tests, measure performance

Steps execute sequentially within their stage. Each step must complete successfully for the next step to begin.

Completion Semantics

FirmwareCI uses strict hierarchical completion logic ensuring test integrity.

Step Completion

A step completes successfully when its underlying command or operation returns a zero exit code. Non-zero exit codes indicate failure.

Success criteria vary by command:

  • cmd: Command exits with code 0
  • ping: Target responds within timeout
  • copy: Files transfer successfully
  • chipsec: Scan completes and expectations match
  • cpustats: CPU metrics meet defined expectations

Stage Completion

A stage completes successfully only when all constituent steps execute without failure. If any step fails, the stage immediately fails.

Stage failure behavior:

  • Mark stage as failed
  • Skip remaining steps in the stage
  • Continue to next stage

This fail-fast behavior prevents unnecessary execution while allowing subsequent stages to run for diagnostic purposes.

Test Completion

A test completes successfully only when every stage executes without failure. Partial success is not recognized - if any stage fails, the entire test fails.

Test result determination:

  • All stages succeed → Test passes
  • Any stage fails → Test fails

Test Completion - Successful Test Completion - Successful

Failure Propagation

When a step fails, FirmwareCI implements fail-fast behavior within stages while continuing test execution.

Within-Stage Failure

stages:
  - name: Boot Test
    steps:
      - cmd: ping              # Succeeds
      - cmd: cmd               # Fails - exit code 1
      - cmd: cmd               # Skipped due to previous failure

The first failure marks the stage as failed and skips remaining steps in that stage.

Test Completion with failed test step Test Completion with failed test step

Cross-Stage Failure

stages:
  - name: Stage 1
    steps:
      - cmd: ping              # Fails
  - name: Stage 2
    steps:
      - cmd: cmd               # Still executes

Even though Stage 1 failed, Stage 2 executes. This allows collecting diagnostic information from later stages. The test result is permanently failed regardless of Stage 2’s outcome.

Execution Flow

Standard Flow

  1. Start: System begins with the first stage in the test file
  2. Execute steps: Within the stage, steps execute sequentially
  3. Step success: If all steps succeed, proceed to next stage
  4. Step failure: If any step fails, skip remaining steps and proceed to next stage
  5. Continue: Repeat until all stages have been attempted
  6. Report: Generate test report with pass/fail status

With Pre/Post Stages

name: Complete Test
description: Full test lifecycle

pre-stage:
  - cmd: dutctl
    name: Flash firmware

stages:
  - name: Boot Test
    steps: [...]

  - name: Security Scan
    steps: [...]

post-stage:
  - cmd: dutctl
    name: Power off DUT

Execution order:

  1. Pre-stage operations
  2. All test stages
  3. Post-stage operations (always run, even if tests fail)

Post-stages always execute to ensure proper cleanup and DUT state reset.

Templating

Tests use template variables to access dynamic data:

Input templates: [[input.Binary]] - Data provided when triggering the workflow

Attribute templates: [[attributes.Host]] - DUT-specific configuration

Storage templates: [[storage.tools]] - References to storage items

Secret templates: [[secrets.APIToken]] - Sensitive data from workflow secrets

YAML anchors: Define reusable configuration blocks under defaults keyword

defaults:
  transport: &transport
    proto: ssh
    options:
      host: "[[attributes.Host]]"
      user: root

stages:
  - name: Test
    steps:
      - cmd: cmd
        transport: *transport
        parameters:
          executable: uname
          args: ["-a"]

For comprehensive templating syntax, see Templating & Variables.

Next Steps