Test Step Commands

This page describes the structure of test files, stages, and test steps in FirmwareCI.

Test Step Structure

Each test step in a stage has this structure:

Attribute	Type	Required	Description
`cmd`	`string`	Yes	Identifies which test step to execute
`name`	`string`	Yes	Uniquely identifies the step for reports and logs
`transport`	`TransportObject`	Yes	Defines how the command is executed
`parameters`	`object`	Yes	Command-specific parameters
`options`	`OptionsObject`	No	Additional command options

Example

name: "Firmware Test Suite"
description: "Tests firmware functionality and boot process"
defaults:
  transport: &default_transport
    proto: ssh
    options:
      host: "[[.Host]]"
      user: root
      identity_file: /root/.ssh/id_ecdsa

tools:
  - "flasher"
  - "serial_monitor"

pre-stage:
  name: "Setup"
  steps:
    - cmd: dutctl
      name: "Power On"
      transport: *default_transport
      parameters:
        command: "power"
        args: ["on"]

stages:
  - name: "Boot Test Stage"
    steps:
      - cmd: firmware-version
        name: "Check BIOS Version"
        transport: *default_transport
        parameters:
          tool_path: "/usr/sbin/fw_version"
          format: "hex"
          expect:
            version: "0xA5"
      
      - cmd: cmd
        name: "Verify Boot Status"
        transport: *default_transport
        parameters:
          executable: "systemctl"
          args: ["is-system-running"]
          expect:
            - regex: "running"

post-stage:
  name: "Cleanup"
  steps:
    - cmd: dutctl
      name: "Power Off"
      transport: *default_transport
      parameters:
        command: "power"
        args: ["off"]

Transport

The transport object defines how test steps are executed, either locally or on a remote machine. This configuration ensures proper command execution and authentication.

Attribute	Type	Required	Description
`proto`	`string`	Yes	Protocol for command execution: `local` (local execution) or `ssh` (remote execution)
`options`	`TransportOptions`	Required for SSH	Protocol-specific configuration options

Transport Options

When using the SSH protocol (proto: "ssh"), the following options are available:

Attribute	Type	Required	Default	Description
`host`	`string`	Yes		Hostname or IP address of remote machine
`user`	`string`	Yes		Username for authentication
`port`	`integer`	No	`22`	SSH port number
`password`	`string`	One of these required*		User’s password
`identity_file`	`string`	One of these required*		Path to SSH private key

* You must provide either password or identity_file, but not both.

Protocol-Specific Requirements

Local Protocol

When using proto: "local":

No options are required or allowed
Commands execute on the local machine

SSH Protocol

When using proto: "ssh":

The options object is required
Must provide host and user
Must provide either password or identity_file
Cannot provide both password and identity_file

Examples

Local Execution

transport:
  proto: local

SSH with Password

transport:
  proto: ssh
  options:
    host: "192.168.1.100"
    user: "admin"
    password: "secretpassword"

SSH with Identity File

transport:
  proto: ssh
  options:
    host: "[[.Host]]"
    user: "root"
    identity_file: "/root/.ssh/id_ecdsa"
    port: 2222  # Optional custom port

Options

The options object provides common configuration settings that can be applied to any test step. These settings control the test step’s execution behavior.

Attribute	Type	Required	Default	Description
`timeout`	`string`	No	`6h00m00s`	Maximum duration for test step execution. If the step takes longer than this timeout, it will fail. Format: hours(h), minutes(m), seconds(s)

Timeout Format

The timeout value must follow this pattern: ^((\\d+h)?(\\d+m)?(\\d+s)?)?$

Valid examples:

"30s" - 30 seconds
"5m" - 5 minutes
"2h30m" - 2 hours and 30 minutes
"1h30m45s" - 1 hour, 30 minutes, and 45 seconds

Examples

Basic Timeout

options:
  timeout: "30s"

Complex Timeout

options:
  timeout: "1h30m45s"

Complete Test Step with Options

cmd: firmware-version
name: "Check BIOS Version"
transport:
  proto: ssh
  options:
    host: "[[.Host]]"
    user: root
    identity_file: /root/.ssh/id_ecdsa
parameters:
  tool_path: "/usr/sbin/fw_version"
  format: "hex"
  expect:
    version: "0xA5"
options:
  timeout: "5m"  # Step will fail if not completed within 5 minutes

Teststep Commands

Binarly

The Binarly test step executes a scan using the Binarly service to analyze firmware binaries.

Attribute	Type	Required	Description
`cmd`	`const`	Yes	Must be `binarly report`
`name`	`string`	Yes	Uniquely identifies the test step in reports and logs
`parameters`	`BinarlyParameters`	Yes	Binarly-specific parameters
`options`	`OptionsObject`	No	Additional command options

Binarly Parameters

Attribute	Type	Required	Description
`url`	`string`	Yes	URL of the Binarly endpoint
`token`	`string`	Yes	Authentication token for Binarly service
`file_path`	`string`	Yes	Path to the binary file to be scanned

Example

cmd: binarly report
name: "Scan BIOS for vulnerabilities"
parameters:
  url: "https://api.binarly.io/v1"
  token: "your-binarly-token"
  file_path: "/path/to/firmware.bin"
options:
  timeout: "30m"  # Scan might take a while

ChipSec

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`chipsec`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`ChipSecOptions`			Options passed into ChipSec
`input`	`ChipSecInputOptions`		`parameters`	`chipsec` Input Options

ChipSec Options

ChipSec only supports one set of options.

ChipSec Input Options

Attribute	Type	Default	Parent	Description
`pch`	`string`		`input`	PCH parameter that gets passed into `ChipSec` as `--pch` parameter
`modules`	`[]string`		`input`	List of modules that get executed from the `ChipSec` test suite
`nix_os`	`bool`	`false`	`input`	Specify if the `ChipSec` tooling gets executed on a NixOS platform

ChipSec Example

Copy

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`copy`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`CopyOptions`			Options passed into the `copy` cmd
`input`	`CopyInputOptions`		`parameters`	`copy` Input Options

Copy Options

Copy Input Options

Attribute	Type	Default	Parent	Description
`source`	`string`		`input`	Source Location of the file to copy to the DUT.
`destination`	`string`		`input`	Destination Location of the file to copy to the DUT
`recursive`	`bool`	`false`	`input`	Copy false recursively. Available options: `false`, `true`.

Copy Example

CPULoad

CPULoad is a subcommand of the System-Suite. Check out the System-Suite for full documentation.

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`cpuload`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`CPULoadOptions`			Options passed into the `cpuload` cmd
`input`	`CPULoadInputOptions`		`parameters`	`cpuload` Input Options
`expect`	`CPULoadExpectOptions`		`parameters`	`cpuload` Input Options

CPULoad Options

CPULoad Input Options

Attribute	Type	Parent	Description
`tool_path`	`string`	`input`	Path to the tool on the DUT
`args`	`[]string`	`input`	Array of `strings` with arguments passed into the `cpuload` tool. Available options: `XXXTODOXXX`
`cpus`	`[]integer`	`input`	Array of CPUs that should be put under load.
`duration`	`string`	`input`	Duration of the load. Format: `^((\\d+h)?(\\d+m)?(\\d+s)?)?$`

CPULoad Expect Options

Attribute	Type	Default	Parent	Description
`general`	`[]GeneralObject`		`expect`	GeneralObject Description
`individual`	`[]IndividualObject`		`expect`	IndividualObject Description

General Object

Attribute	Type	Default	Parent	Description
`option`	`string`		`general`	Name of the option
`value`	`string`		`general`	Value of the option

Individual Object

Attribute	Type	Parent	Description
`cpu`	`[]integer`	`individual`	Array of CPUs where the option should be applied to
`option`	`string`	`individual`	Name of the option
`value`	`string`	`individual`	Value of the option

CPULoad Example

CPUSet

cpuset is part of the System-Suite. Check out the documentation of the System-Suite for more information.

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`cpuset`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`CPUSetOptions`			Options passed into the `cpuset` cmd
`input`	`CPUSetInputOptions`		`parameters`	`cpuset` Input Options

CPUSet Options

CPUSet Input Options

Attribute	Type	Parent	Description
`tool_path`	`string`	`input`	Path to the tool on the DUT
`command`	`string`	`input`	Command to run. Available options: `core`, or `profile`.
`args`	`[]string`	`input`	Array of `strings` with arguments passed into the `cpuset` tool. Available options: `XXXTODOXXX`
`cores`	`[]integer`	`input`	Array of Cores where the command should be applied to.
`duration`	`string`	`input`	Duration of the load. Format: `^((\\d+h)?(\\d+m)?(\\d+s)?)?$`

CPUSet Example

CPUStats

cpustats is part of the System-Suite. Check out the documentation of the System-Suite for more information.

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`cpustats`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`CPUStatsOptions`			Options passed into the `cpustats` cmd
`input`	`CPUStatsInputOptions`		`parameters`	`cpustats` input options
`expect`	`CPUStatsExpectOptions`		`parameters`	`cpustats` expect options

CPUStats Options

CPUStats Input Options

Attribute	Type	Default	Parent	Description
`tool_path`	`string`		`input`	Path to the tool on the DUT
`interval`	`string`		`input`	Interval to check the CPU Stats

CPUStats Expect Options

Attribute	Type	Default	Parent	Description
`general`	`[]GeneralObject`		`expect`	GeneralObject Description
`individual`	`[]IndividualObject`		`expect`	IndividualObject Description

CPUStats Example

DUTCTL

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`dutctl`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`DUTCTLOptions`			Options passed into the `dutctl` cmd
`input`	`DUTCTLInputOptions`		`parameters`	`dutctl` input options

DUTCTL Options

DUTCTL Input Options

Attribute	Type	Parent	Description
`host`	`string`	`input`	Hostname or IP address of the device.
`uart`	`integer`	`input`	UART number to use for serial communication.
`command`	`string`	`input`	Command to execute on the device. Available commands: `power`, `flash`, `serial`.
`args`	`[]string`	`input`	Arguments for the command. Arguments depend on the command that is executed.
`input`	`string`	`input`	Input for the command, used only if command is set to `serial`.

DUTCTL Expect Options

Attribute	Type	Default	Parent	Description
`regex`	`string`		`expect`	Regular expression to match against the output of the device for the `serial` command.

Firmware Version

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`firmware version`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`FirmwareVersionOptions`			Options passed into the `firmware version` cmd
`input`	`FirmwareVersionInputOptions`		`parameters`	`firmware version` input options
`expect`	`FirmwareVersionExpectOptions`		`parameters`	`firmware version` expect options

Firmware Version Options

Firmware Version Input Options

Attribute	Type	Default	Parent	Description
`tool_path`	`string`		`input`	Path to the tool used to retrieve the firmware version.
`format`	`string`		`input`	Format of the tool’s output. Available options: `number`, `hex`, `pair`, `triplet`, `quad`.

Firmware Version Expect Options

Attribute	Type	Default	Parent	Description
`version`	`string`		`expect`	Expected firmware version in the specified format.

FWHunt

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`fwhunt`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`FWHuntOptions`			Options passed into the `fwhunt` cmd
`input`	`FWHuntInputOptions`		`parameters`	`fwhunt` input options

FWHunt Options

FWHunt Input Options

Attribute	Type	Parent	Description
`rules_dirs`	`[]string`	`input`	List of directories to search for rules. If not specified, the default rules directory will be used.
`rules`	`[]string`	`input`	List of rules to run.
`report_only`	`boolean`	`input`	Only report findings, do not interpret the output.

FWTS

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`fwts`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`FWTSOptions`			Options passed into the `fwts` cmd
`input`	`FWTSInputOptions`		`parameters`	`fwts` input options

FWTS Options

FWTS Input Options

Attribute	Type	Default	Parent	Description
`flags`	`[]string`		`input`	List of flags to pass to `fwts`.
`report_only`	`boolean`		`input`	Only report findings, do not interpret the output.

Ping

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`ping`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`PingOptions`			Options passed into the `ping` cmd
`input`	`PingInputOptions`		`parameters`	`ping` input options

Ping Options

Ping Input Options

Attribute	Type	Default	Parent	Description
`host`	`string`		`input`	Hostname or IP address used for the ping.
`port`	`integer`	`22`	`input`	Port to use for the ping.
`expect`	`PingExpectOptions`		`parameters`	`ping` expect options

Ping Expect Options

Attribute	Type	Default	Parent	Description
`should_fail`	`boolean`	`false`	`expect`	If set to true, the expected behavior is that the ping won’t work. Otherwise, a successful ping is expected.

Qemu

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`qemu`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`QemuOptions`			Options passed into the `qemu` cmd
`input`	`QemuInputOptions`		`parameters`	`qemu` input options

Qemu Options

Qemu Input Options

Attribute	Type	Parent	Description
`executable`	`string`	`input`	The executable that should be executed.
`firmware`	`string`	`input`	The firmware that should be used.
`image`	`string`	`input`	The image that should be used.
`logfile`	`string`	`input`	The logfile that should be used.
`nproc`	`integer`	`input`	The number of processors that should be used.
`mem`	`integer`	`input`	The amount of memory that should be used.
`steps`	`QemuStepsOptions`	`input`	Steps to interact with during test execution.

Qemu Steps Options

Attribute	Type	Parent	Description
`send`	`string`	`input`	A string that should be sent as input.
`timeout`	`string`	`input`	Timeout for the interaction. Format: h for hours, m for minutes, s for seconds (e.g., 1h30m20s).
`expect`	`QemuExpectOptions`	`input`	Expectation for the interaction output.

Qemu Expect Options

Attribute	Type	Default	Parent	Description
`regex`	`string`		`expect`	Regular expression that should be matched against the output of the interaction.

Robot Test Framework

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`robot`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`RobotOptions`			Options passed into the `robot` cmd
`input`	`RobotInputOptions`		`parameters`	`robot` input options

Robot Options

Robot Input Options

Attribute	Type	Parent	Description
`file_path`	`string`	`input`	Path to the robot test file.
`args`	`[]string`	`input`	Arguments to pass to the robot test file.
`report_only`	`boolean`	`input`	If true, the output won’t be validated.

S0ix-Selftest

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`s0ix-selftest`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`S0ixSelftestOptions`			Options passed into the `s0ix-selftest` cmd
`input`	`S0ixSelftestInputOptions`		`parameters`	`s0ix-selftest` input options

S0ix-Selftest Options

S0ix-Selftest Input Options

Attribute	Type	Default	Parent	Description
`tool_path`	`string`		`input`	Path to the tool used to perform the S0ix self-test.

SecureBoot Management

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`Secure Boot Management`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`SecureBootOptions`			Options passed into the `Secure Boot Management` cmd
`input`	`SecureBootInputOptions`		`parameters`	`Secure Boot Management` input options

SecureBoot Options

SecureBoot Input Options

Attribute	Type	Parent	Description
`command`	`string`	`input`	Command to execute. Can be one of: `status`, `enroll-key`, `rotate-key`, `reset`, `custom-key`.
`tool_path`	`string`	`input`	Path to the tool that should be executed to run Secure Boot commands.
`hierarchy`	`string`	`input`	Hierarchy to use for the key. Can be one of: `db`, `dbx`, `KEK`, `PK`.
`key_file`	`string`	`input`	Path to the key file used for `enroll-key` and `rotate-key` commands.
`custom_key_file`	`string`	`input`	Path to the custom key file used for `custom-key` command.
`cert_file`	`string`	`input`	Path to the certificate file used for `enroll-key` and `rotate-key` commands.
`signing_key_file`	`string`	`input`	Path to the signing key file used for `enroll-key` and `rotate-key` commands.
`signing_cert_file`	`string`	`input`	Path to the signing certificate file used for `enroll-key` and `rotate-key` commands.
`append`	`boolean`	`input`	Append the key to the existing key database.
`expect`	`SecureBootExpectOptions`	`parameters`	Expectations for the command execution.

SecureBoot Expect Options

Attribute	Type	Parent	Description
`should_fail`	`boolean`	`expect`	If set to true, the test step will fail if the command does not fail.
`secure_boot`	`boolean`	`expect`	If set to true, the test step will fail if Secure Boot is not enabled.
`setup_mode`	`boolean`	`expect`	If set to true, the test step will fail if Secure Boot is not in setup mode.

Sleep

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`sleep`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`SleepOptions`			Options passed into the `sleep` cmd
`input`	`SleepInputOptions`		`parameters`	`sleep` input options

Sleep Options

Sleep Input Options

Attribute	Type	Default	Parent	Description
`duration`	`string`		`input`	Duration to sleep. Format: 1h2m3s. Units: h, m, s.

SSHCmd

sshcmd is the generic command to execute commands.

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`sshcmd`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`SSHCmdOptions`			Options passed into the `sshcmd` cmd
`input`	`SSHCmdInputOptions`		`parameters`	`sshcmd` input options
`expect`	`SSHCmdExpectOptions`		`parameters`	`sshcmd` expect options

SSHCmd Options

SSHCmd Input Options

Attribute	Type	Parent	Description
`executable`	`string`	`input`	Executable that should be executed. Full Path, or need to be part of `$PATH`
`args`	`[]string`	`input`	Array of arguments passed into the `executable`
`working_dir`	`string`	`input`	`PWD` the executable will be executed.
`report_only`	`boolean`	`input`	`sshcmd` is allowed to fail, it only reports the output.

SSHCmd Expect Options

Attribute	Type	Default	Parent	Description
`regex`	`string`		`expect`	Regex that matches on the expected output value.

SSHCmd Example

SysBench

Attribute	Type	Default	Parent	Description
`cmd`	`const`	`sysbench`
`name`	`string`			Name to uniquely identify the teststep. This name will show up in reports, logs and the WebUI
`parameters`	`SysBenchOptions`			Options passed into the `sysbench` cmd
`input`	`SysBenchInputOptions`		`parameters`	`sysbench` input options

SysBench Options

SysBench Input Options

Attribute	Type	Default	Parent	Description
`args`	`[]string`		`input`	Arguments to pass to `sysbench`.
`expect`	`[]ExpectOptions`		`input`	Expectations to check against the output of `sysbench`.