Writing a Simple Boot Test

In this guide, we will cover how to modify the initial set of files provided by the Initial Setup guide. By following these instructions, you will get a better idea of how to customize your testing environment to better suit your needs.

Let’s implement a boot test that power cycles your board.

Note: The names of the DUT and workflows are autogenerated, so your configuration names may vary but will follow a similar pattern.

Prerequisites

• Ensure you have the fwci CLI tool installed on your system.
• A git repository where you have already set up and synchronized the exemplary .firmwareci infrastructure. Refer to the Initial Setup for detailed instructions on setting up the initial configuration.

Validation

The fwci CLI provides a command to validate the content of the .firmwareci directory. Simply execute the command either in the repository’s parent directory or provide any path to it as an argument.

$ fwci validate

It will provide detailed information file-by-file in case any validation fails:

Example of a failed validation:

$ fwci validate

- duts
  - all validations passed
- storage
  - all validations passed
- workflows
  - fwci-crystal-core
    - workflow.yaml: invalid YAML:
      - (root): runs-on is required
      - (root): Additional property works-on is not allowed

As we make changes, ensure to run the validation from time to time to make sure everything is in order.

Modifying the Exemplary (DUT)

First, let’s configure the DUT with all the necessary information we might need for our test case. Open the DUT configuration dut.yaml.

The most critical fields have been pre-populated with placeholder information:

# Example Device-under-Test Configuration for fwci-crystal-core
name: Example-DUT
label: dut-example-label
attributes:
  Host: "host.local"
  Flasher: "flasher.local"

For a comprehensive explanation, refer to the DUT Configuration documentation.

Suppose you have an ARM-based board with the hostname arm-board.lan, connected to a DUT control interface (DUTCTL) with the hostname arm-board-dutctl.lan.

Update the dut.yaml with the following details:

# Example Device-under-Test Configuration for ARM-based board
name: ARM-Board-DUT
label: arm-board
attributes:
    Host: "arm-board.lan"
    Flasher: "arm-board-dutctl.lan"

Pre-Stage and Post-Stage Configuration

To ensure a consistent setup and teardown process for your DUT, you can use pre-stage and post-stage configuration files. These files should be placed in the DUTs directory and will be executed before and after the tests, respectively.

• Pre-Stage (pre.yaml): This file contains the necessary steps to prepare the DUT before the tests are executed. It ensures that the DUT is in the correct state and ready for testing.
• Post-Stage (post.yaml): This file includes the steps to clean up and reset the DUT after the tests have been completed. It ensures that the DUT is returned to a known state, ready for the next set of tests.

By using these configuration files, you can maintain a standardized process for preparing and cleaning up your DUT, leading to more reliable and repeatable test results. For a comprehensive explanation, refer to the DUT Configuration documentation.

Let’s modify both files for our use case. The pre-stage file should start by ensuring the DUT is powered off, flash the firmware binary, and then turn it on. The post-stage should clean up by powering the device down.

pre.yaml

pre-stage:
  - cmd: dutctl
    name: Shutdown the DUT.
    parameters:
      host: "[[attributes.Flasher]]"
      command: power
      args: ["off"]

  - cmd: dutctl
    name: Flash the firmware binary.
    parameters:
      host: "[[attributes.Flasher]]"
      command: flash
      args: [write, "[[defaults.Binary]]"]


  - cmd: dutctl
    name: Turn the DUT on
    parameters:
      host: "[[attributes.Flasher]]"
      command: power
      args: ["hardreset"]
    options:
      timeout: 2m

post.yaml

post-stage:
  - cmd: dutctl
    name: Clean up. Shutdown the DUT.
    parameters:
      host: "[[attributes.Flasher]]"
      command: power
      args: ["off"]

Templating Excourse

Templating allows for dynamic generation of tests. In the provided pre.yaml and post.yaml examples, we utilize the [[attributes.Flasher]] placeholder to dynamically reference the flasher address defined in the DUT configuration.

For a comprehensive explanation, refer to the Templating documentation.

Modifying the Workflow & Adding a Test

Workflow

The workflow file is located inside the workflows directory, with its main configuration named workflow.yaml. Let’s take a look at the placeholder values.

# Example Workflow Configuration for fwci-crystal-core
name: fwci-crystal-core
description: "This is a sample workflow configuration"
runs-on: dut-example-label

For a comprehensive explanation, refer to the Workflow Configuration documentation.

Let’s change the configuration so its tests will run on our modified DUT with the label arm-board.

# Example Workflow Configuration for ARM-based-board-workflow
name: ARM-based-board-workflow
description: "This workflow will run tests on an exemplary ARM-based board"
works-on: arm-board

Tests

The final step is to write additional tests for our modified workflow. Let’s start by examining the existing test and then add a new boot test.

For a comprehensive explanation, refer to the Test Files documentation.

Examining the Existing Test

# Example Test Configuration for fwci-crystal-core
name: Example Test
description: This is a sample test configuration

stages:
- name: Example Stage
  steps:
  - cmd: cmd
    name: Example Step
    transport:
      proto: local
    parameters:
      executable: echo
      args: ["Hello, World!"]

This test is designed to succeed regardless of the DUT configuration. It executes a simple echo Hello, World! command within the local testing environment, without interacting with the DUT.

Adding a Boot Test

To add a new boot test, create a boot.yaml file in the tests directory.

Here’s an example of a boot test configuration:

# Boot Test Configuration for ARM-based board
name: Boot Test
description: This test verifies that the ARM-based board boots correctly.

stages:
  - name: Boot Stage
    steps:
      - cmd: ping
        name: Wait for SSH service
        options:
          timeout: 2m
        parameters:
          host: "[[attributes.Host]]"

      - cmd: cmd
        name: Run echo "Hello World" on the device
        transport:
          proto: ssh
          options:
            host: "[[attributes.Host]]"
            user: root
            password: root
        parameters:
          executable: echo
          args: ["Hello, World!"]

This boot test verifies the successful booting of the DUT. It begins by executing a ping command to the DUT’s designated host address, followed by an echo command executed on the DUT itself.

Note: Remember that the entire setup and teardown process is managed by the pre-stage and post-stage configurations of the DUT. However, their steps will also be included in the test run.

Conclusion

With all the modifications in place, it’s essential to validate the updated configuration to ensure everything is set up correctly. Run the following command to perform a final validation:

$ fwci validate

- duts
  - all validations passed
- storage
  - all validations passed
- workflows
  - fwci-crystal-core

$ tree .firmwareci

.firmwareci/
├── duts
│   └── dut-fwci-crystal-core
│       ├── dut.yaml
│       ├── post.yaml
│       └── pre.yaml
├── README.md
├── storage
└── workflows
  └── fwci-crystal-core
    ├── tests
    │   ├── boot.yaml
    │   └── test.yaml
    └── workflow.yaml

If the validation passes without any errors, you can proceed to commit the changes to your git repository. This will apply the new configurations and tests, making them available for your testing environment.

$ git add .
$ git commit -m "CI: updated DUT configuration, workflow, and added boot test"
$ git push

By following these steps, you ensure that your testing environment is correctly configured and ready for use. Happy testing!