Test Files

The test file defines the configuration and execution flow of a test. Below are the key components:

Required Fields

Attribute	Type	Description
name	`string`	Name of the test
description	`string`	Description of the test
stages	`[]stage`	List of stages defining test operations

Optional Fields

Attribute	Type	Description
pre-stage	`stage`	Overrides default DUT pre-stage configuration
post-stage	`stage`	Overrides default DUT post-stage configuration
defaults	`object`	Defines defaults to simplify the test file
binary-not-required	`boolean`	If true, test can run without a binary
on-error	`string`	Error handling behavior: `strict` (default) or `continue`. See On-Error Behavior

Stage Schema

Each stage in the stages array follows this structure:

Attribute	Type	Description
name	`string`	Name of the stage
steps	`array`	List of test step commands (see Commands)
on-error	`string`	Error handling for all steps in this stage: `strict` (default) or `continue`. Overrides test-level setting.

Step Options

Each step can include an options block with these fields:

Attribute	Type	Description
timeout	`string`	Maximum execution time (e.g., `5m`, `30s`)
on-error	`string`	Error handling for this step: `strict` (default) or `continue`. Overrides stage and test-level settings.
path	`string`	Working directory for the command

Naming conventions

Steps and stages should be named in a way that clearly describes the operation or command being executed. This helps in identifying the purpose of each step and stage, making it easier to understand the test file.

Additionally, stage names must be unique within a test file — no two stages may share the same name. Likewise, within a single stage all step names must be unique — no two steps in the same stage may share the same name.

Example Test Files

name: Boot Test (Network)
description: Flash the provided firmware binary and conduct a boot test.

defaults:
  transport: &transport
    proto: ssh
    options:
      host: "[[attributes.Host]]"
      user: root
      # Uses auto-discovery of organization SSH keys

stages:
  - name: Boot Test
    steps:
      - cmd: cmd
        name: Cat OS Version
        transport: *transport
        parameters:
          executable: cat
          args: ["/etc/os-release"]

      - cmd: cmd
        name: Power off device
        transport: *transport
        parameters:
          executable: shutdown
          args: ["now"]

name: Boot Test (Serial)
description: Flash the provided firmware binary and conduct a boot test.

defaults:
  transport: &transport
    proto: ssh
    options:
      host: "[[attributes.Host]]"
      user: root
      identity_file: "[[ssh-keys.device_key]]" # Reference organization SSH key

pre-stage:
  - cmd: dutctl
    name: Shutdown the DUT.
    parameters:
      host: "[[attributes.Pi]]"
      command: power
      args: ["off"]

  - cmd: dutctl
    name: Flash the provided binary.
    parameters:
      host: "[[attributes.Pi]]"
      command: flash
      args: [write, "[[input.Binary]]"]

  - cmd: sleep
    name: Wait for the DUT to settle
    parameters:
      duration: 15s

  - cmd: dutctl
    name: Turn the DUT on
    parameters:
      host: "[[attributes.Pi]]"
      command: power
      args: ["hardreset"]
    options:
      timeout: 2m

  - cmd: dutctl
    name: Check Serial
    parameters:
      host: "[[attributes.Pi]]"
      command: serial
      expect:
        - regex: "XhcClearBiosOwnership: called to clear BIOS ownership"
    options:
      timeout: 5m

stages:
  - name: Boot Test
    steps:
      - cmd: cmd
        name: Cat OS Version
        transport: *transport
        parameters:
          executable: cat
          args: ["/etc/os-release"]

      - cmd: cmd
        name: Power off device
        transport: *transport
        parameters:
          executable: shutdown
          args: ["now"]

Templates in Test Files

Test files often reference two types of templates:

DUT Attribute Templates: These provide configuration values for the Device Under Test (DUT), such as hostnames, hardware settings, or transport options.
Job Input Templates: These supply dynamic values from job requests, like firmware binaries or test parameters.

See Templating References for more information on templates.