Test Execution Model

Overview

FirmwareCI test files are structured into stages containing steps. This hierarchical structure enables organized, sequential execution with clear success criteria and failure handling.

Test Structure

Hierarchy

Test File
├── Pre-Stage (optional)
├── Stage 1
│   ├── Step 1
│   ├── Step 2
│   └── Step 3
├── Stage 2
│   ├── Step 1
│   └── Step 2
└── Post-Stage (optional)

The Pre-stage (Setup) prepares the DUT before the stages, and the Post-stage (Teardown) cleans up after them. Both are optional and usually come from the DUT’s pre and post files (see Execution Flow).

Stages

Stages are named groups of related operations executed sequentially. Each stage contains one or more steps that accomplish a specific testing objective.

Stages and Steps Stages and Steps

Example stages:

  • Boot Test: Verify device boots and network connectivity
  • Firmware Validation: Run security scans and integrity checks
  • Performance Benchmarking: Execute performance tests and collect metrics
  • Stress Testing: Apply load and verify stability

Stages execute in definition order. If a step fails, its stage fails and the following stages are skipped (see When a Step Fails).

Steps

Steps are individual test operations executed within a stage. FirmwareCI offers a comprehensive set of Test Step Commands ranging from simple bash commands to complex integrations like the Robot Test Framework.

Common step types:

  • Command execution: Run shell commands on the DUT
  • File operations: Copy files, create archives
  • Network operations: Ping devices, check connectivity
  • Firmware analysis: Scan with ChipSec, Binarly, FWHunt
  • Hardware control: Power cycle, flash firmware, access serial
  • Benchmarking: Run CPU tests, measure performance

Steps execute sequentially within their stage. By default, each step must succeed for the next step to begin.

Step Success Criteria

A step succeeds when its underlying command or operation returns a zero exit code. Non-zero exit codes indicate failure.

Success criteria vary by command:

  • cmd: Command exits with code 0
  • ping: Target responds within timeout
  • copy: Files transfer successfully
  • chipsec: Scan completes and expectations match
  • cpustats: CPU metrics meet defined expectations

When a Step Fails

By default, a test stops at the first failed step:

  1. The step fails, so its stage and the test fail.
  2. The remaining steps of the stage are skipped.
  3. All following stages are skipped.
  4. The Post-stage (Teardown) still runs, for cleanup.
stages:
  - name: Boot Test
    steps:
      - cmd: ping              # Succeeds
      - cmd: cmd               # Fails: exit code 1
      - cmd: cmd               # Skipped
  - name: Security Scan
    steps:
      - cmd: chipsec           # Skipped

Test Completion with failed test step Test Completion with failed test step

A test passes only if every stage passes. There is no partial success.

Test Completion - Successful Test Completion - Successful

Changing the Default

Two options change what happens after a failed step:

  • on-failure: continue runs the remaining steps of the same stage (see On-Failure Behavior).
  • A failure stage runs only after a failure, for example to collect data from the DUT (see Run Conditions).

On-Failure Behavior

on-failure: continue runs the remaining steps of a stage after one of them failed. The stage still fails, so the following stages are skipped:

stages:
  - name: Boot Test
    on-failure: continue
    steps:
      - cmd: ping              # Fails
      - cmd: cmd               # Still runs
  - name: Security Scan
    steps:
      - cmd: chipsec           # Skipped: Boot Test failed
ValueAfter a failed step
strict(Default) The remaining steps of the stage are skipped.
continueThe remaining steps of the stage run. The stage still fails.

Where to Set It

Set on-failure at one of these levels. The most specific level wins:

  1. Test level: at the top of the test file, for every stage
  2. Stage level: in a stage, for its steps
  3. Step level: in the step’s options block, for this step only

The Pre-stage (Setup) and the Post-stage (Teardown) do not inherit the test-level value. Set it in their own steps, or at the top of a DUT pre/post file.

on-failure was called on-error before. The old name still works, but it is deprecated; if both are set, on-failure wins.

Run Conditions

run-condition: failure turns the last stage of a test into a failure stage. It runs only if the test failed:

stages:
  - name: Boot
    steps: [...]

  - name: Collect logs
    run-condition: failure
    steps: [...]
WhenThe failure stage
The test passesis skipped
A stage failedruns, then the Post-stage runs
The Pre-stage failedis skipped: the test never started, so only the Post-stage runs

Rules:

  • Only the last stage of the test file takes run-condition, and its only value is failure. Validation rejects it on any other stage. The Pre-stage and the Post-stage do not take it.
  • If a step of the failure stage fails, the remaining steps of the failure stage are skipped, as in every stage. on-failure: continue changes this, as in every stage.
  • An abort or a job timeout stops every stage, also the failure stage and the Post-stage.

Which Step Failed

After the first failed step, {{ FAILED_STAGE }} and {{ FAILED_STEP }} name the stage and the step that failed, with spaces and special characters removed like every variable value. While nothing failed, both are empty.

This failure stage collects a BMC dump named after the failed stage, before the Post-stage powers the DUT off. on-failure: continue runs the archive step even if the dump command fails:

stages:
  - name: Boot
    steps: [...]

  - name: Collect BMC dump
    run-condition: failure
    on-failure: continue
    steps:
      - cmd: shell
        name: Create BMC dump
        parameters:
          command: dreport -d /tmp -n ffdc-{{ FAILED_STAGE }}
        transport:
          proto: ssh
          options:
            host: "[[attributes.BMCHost]]"
            user: root
      - cmd: archive
        name: Attach BMC dump
        parameters:
          path: /tmp/ffdc-{{ FAILED_STAGE }}.tar.xz
        transport:
          proto: ssh
          options:
            host: "[[attributes.BMCHost]]"
            user: root

Both Together

on-failure: continue in a normal stage and a failure stage work together:

stages:
  - name: Boot Test
    on-failure: continue
    steps:
      - cmd: ping              # Fails
      - cmd: cmd               # Still runs
  - name: Security Scan
    steps:
      - cmd: chipsec           # Skipped: Boot Test failed
  - name: Collect BMC dump
    run-condition: failure
    on-failure: continue
    steps: [...]               # Runs

The order is: all steps of Boot Test, Security Scan skipped, all steps of Collect BMC dump, then the Post-stage.

Execution Flow

With Pre/Post Stages

name: Complete Test
description: Full test lifecycle

pre-stage:
  - cmd: dutctl
    name: Flash firmware

stages:
  - name: Boot Test
    steps: [...]

  - name: Security Scan
    steps: [...]

post-stage:
  - cmd: dutctl
    name: Power off DUT

Execution order:

  1. Pre-stage operations (Setup)
  2. The test stages, until one fails
  3. The failure stage, if a stage failed and the test file has one
  4. Post-stage operations (Teardown) - always executed, except after an abort or a job timeout

Post-Stage Guaranteed Execution

The Post-stage (Teardown) always executes, even when:

  • Any test stage fails
  • The Pre-stage fails
  • Steps within stages fail

Only an abort or a job timeout stops it, so it cannot clean up then.

This guarantees proper cleanup and DUT state reset. Use the Post-stage for:

  • Powering off the device
  • Releasing hardware resources
  • Cleaning up temporary files
  • Resetting device state

Note: The Post-stage does not inherit on-failure from the test level. To run every cleanup step even if one fails, set on-failure: continue at the top of the DUT post-stage file:

on-failure: continue    # Ensure all cleanup steps run
post-stage:
  - cmd: dutctl
    name: Power off DUT
    parameters:
      agent: "[[attributes.Agent]]"
      device: "[[attributes.Device]]"
      command: power
      args: ["off"]
  - cmd: cmd
    name: Clean temp files
    parameters:
      executable: rm
      args: ["-rf", "/tmp/test-*"]

The dutctl command name power comes from the agent’s configuration (see Commands).

Templating

Tests use template variables to access dynamic data:

Input templates: [[input.Binary]] - Data provided when triggering the workflow

Attribute templates: [[attributes.Host]] - DUT-specific configuration

Storage templates: [[storage.tools]] - References to storage items

Secret templates: [[secrets.APIToken]] - Sensitive data from workflow secrets

YAML anchors: Define reusable configuration blocks under defaults keyword

defaults:
  transport: &transport
    proto: ssh
    options:
      host: "[[attributes.Host]]"
      user: root

stages:
  - name: Test
    steps:
      - cmd: cmd
        transport: *transport
        parameters:
          executable: uname
          args: ["-a"]

For comprehensive templating syntax, see Templating & Variables.

Next Steps