Test File Structure and Execution

Test File Structure and Execution

Overview

Test files in our system are structured into multiple stages, encapsulating a series of steps that define the specific actions or commands to be executed. This structure facilitates organized, sequential execution of tests, allowing for comprehensive and logical progression through test scenarios.

Stages and Steps

Stages

Each test files consists of multiple stages, which are executed sequentially, one after the other. Each stage contains a list of steps, which are the individual operations or commands executed within that stage.

Stages and Steps Stages and Steps

Steps

A step is a single operation or command executed as part of a stage. Steps within a stage are executed in the order they are defined. FirmwareCI provides a set of [Teststeps] (/refernce/3_commands)that can be used to define the steps in a test file. Steps can be as simple as a single bash command or as complex as executing the Robot Test Framework within FirmwareCI.

Test / Stage / Step Completion

  • Test Completion: A test is considered successful when all stages have been executed successfully.
  • Stage Completion: A stage is considered successful when all steps within that stage have been executed successfully.
  • Step Completion: A step is considered successful when the command or operation it defines is executed successfully.

If a step fails, the system will mark the stage as failed and skip any remaining steps in that stage. The system will then proceed to the next stage in the test file. However, once a step fails and thus the stage is marked as failed, the overall test will be marked as failed.

Test Completion with failed test step Test Completion with failed test step

If all steps with a stage are successful, and all stages are successful, the test will be marked as successful.

Test Completion - Successful Test Completion - Successful

Execution Flow

  1. Start with the first stage: The system begins execution with the first stage in the test file.
  2. Execute steps in sequence: Within the stage, steps are executed in sequence until either all steps are executed successfully, or a step fails.
  3. Proceed or halt:
    • If all steps in the stage complete successfully, the system proceeds to the next stage.
    • If any step within the stage fails, the system marks the stage as failed, skips any remaining steps in that stage, and proceeds to the next stage.
  4. Continue until completion: This process continues until all stages have been attempted.

Templating and Defaults

  • Users can utilize YAML templating (Anchors) and default values to streamline test file creation and enhance reusability.
  • Templating allows for dynamic substitution of values within the test file.
  • For detailed information on YAML syntax, templating, and defaults, refer to the official YAML documentation.
  • FirmwareCI supports all standard YAML syntax and features, including anchors, aliases, and references from Specification 1.2. (August 2021)

This structured approach ensures that test execution is logical, manageable, and consistent with the defined test scenarios. By using stages and steps, users can create detailed and comprehensive test plans that accurately reflect the testing requirements and objectives.