Test Execution Model
Overview
FirmwareCI test files are structured into stages containing steps. This hierarchical structure enables organized, sequential execution with clear success criteria and failure handling.
Test Structure
Hierarchy
The Pre-stage (Setup) prepares the DUT before the stages, and the Post-stage (Teardown) cleans up after them. Both are optional and usually come from the DUT’s pre and post files (see Execution Flow).
Stages
Stages are named groups of related operations executed sequentially. Each stage contains one or more steps that accomplish a specific testing objective.
Example stages:
- Boot Test: Verify device boots and network connectivity
- Firmware Validation: Run security scans and integrity checks
- Performance Benchmarking: Execute performance tests and collect metrics
- Stress Testing: Apply load and verify stability
Stages execute in definition order. If a step fails, its stage fails and the following stages are skipped (see When a Step Fails).
Steps
Steps are individual test operations executed within a stage. FirmwareCI offers a comprehensive set of Test Step Commands ranging from simple bash commands to complex integrations like the Robot Test Framework.
Common step types:
- Command execution: Run shell commands on the DUT
- File operations: Copy files, create archives
- Network operations: Ping devices, check connectivity
- Firmware analysis: Scan with ChipSec, Binarly, FWHunt
- Hardware control: Power cycle, flash firmware, access serial
- Benchmarking: Run CPU tests, measure performance
Steps execute sequentially within their stage. By default, each step must succeed for the next step to begin.
Step Success Criteria
A step succeeds when its underlying command or operation returns a zero exit code. Non-zero exit codes indicate failure.
Success criteria vary by command:
- cmd: Command exits with code 0
- ping: Target responds within timeout
- copy: Files transfer successfully
- chipsec: Scan completes and expectations match
- cpustats: CPU metrics meet defined expectations
When a Step Fails
By default, a test stops at the first failed step:
- The step fails, so its stage and the test fail.
- The remaining steps of the stage are skipped.
- All following stages are skipped.
- The Post-stage (Teardown) still runs, for cleanup.
A test passes only if every stage passes. There is no partial success.
Changing the Default
Two options change what happens after a failed step:
on-failure: continueruns the remaining steps of the same stage (see On-Failure Behavior).- A failure stage runs only after a failure, for example to collect data from the DUT (see Run Conditions).
On-Failure Behavior
on-failure: continue runs the remaining steps of a stage after one of them failed. The stage still fails, so the following stages are skipped:
| Value | After a failed step |
|---|---|
strict | (Default) The remaining steps of the stage are skipped. |
continue | The remaining steps of the stage run. The stage still fails. |
Where to Set It
Set on-failure at one of these levels. The most specific level wins:
- Test level: at the top of the test file, for every stage
- Stage level: in a stage, for its steps
- Step level: in the step’s
optionsblock, for this step only
The Pre-stage (Setup) and the Post-stage (Teardown) do not inherit the test-level value. Set it in their own steps, or at the top of a DUT pre/post file.
on-failure was called on-error before. The old name still works, but it is deprecated; if both are set, on-failure wins.
Run Conditions
run-condition: failure turns the last stage of a test into a failure stage. It runs only if the test failed:
| When | The failure stage |
|---|---|
| The test passes | is skipped |
| A stage failed | runs, then the Post-stage runs |
| The Pre-stage failed | is skipped: the test never started, so only the Post-stage runs |
Rules:
- Only the last stage of the test file takes
run-condition, and its only value isfailure. Validation rejects it on any other stage. The Pre-stage and the Post-stage do not take it. - If a step of the failure stage fails, the remaining steps of the failure stage are skipped, as in every stage.
on-failure: continuechanges this, as in every stage. - An abort or a job timeout stops every stage, also the failure stage and the Post-stage.
Which Step Failed
After the first failed step, {{ FAILED_STAGE }} and {{ FAILED_STEP }} name the stage and the step that failed, with spaces and special characters removed like every variable value. While nothing failed, both are empty.
This failure stage collects a BMC dump named after the failed stage, before the Post-stage powers the DUT off. on-failure: continue runs the archive step even if the dump command fails:
Both Together
on-failure: continue in a normal stage and a failure stage work together:
The order is: all steps of Boot Test, Security Scan skipped, all steps of Collect BMC dump, then the Post-stage.
Execution Flow
With Pre/Post Stages
Execution order:
- Pre-stage operations (Setup)
- The test stages, until one fails
- The failure stage, if a stage failed and the test file has one
- Post-stage operations (Teardown) - always executed, except after an abort or a job timeout
Post-Stage Guaranteed Execution
The Post-stage (Teardown) always executes, even when:
- Any test stage fails
- The Pre-stage fails
- Steps within stages fail
Only an abort or a job timeout stops it, so it cannot clean up then.
This guarantees proper cleanup and DUT state reset. Use the Post-stage for:
- Powering off the device
- Releasing hardware resources
- Cleaning up temporary files
- Resetting device state
Note: The Post-stage does not inherit on-failure from the test level. To run every cleanup step even if one fails, set on-failure: continue at the top of the DUT post-stage file:
The dutctl command name power comes from the agent’s configuration (see Commands).
Templating
Tests use template variables to access dynamic data:
Input templates: [[input.Binary]] - Data provided when triggering the workflow
Attribute templates: [[attributes.Host]] - DUT-specific configuration
Storage templates: [[storage.tools]] - References to storage items
Secret templates: [[secrets.APIToken]] - Sensitive data from workflow secrets
YAML anchors: Define reusable configuration blocks under defaults keyword
For comprehensive templating syntax, see Templating & Variables.
Next Steps
- Learn to write tests in the Writing Tests guide
- Explore Test File Reference for complete schema
- Review Test Step Commands for available operations


