Skip to main content

Execute AIVA tests from a CI/CD pipeline

AIVA test batches can be executed automatically as part of your CI/CD pipeline. This allows you to run visual end‑to‑end tests on every push, on a schedule, or as a step in your release pipeline.

AIVA currently supports CI/CD execution via:

  • GitHub Actions
  • GitLab CI

Both integrations use the same underlying AIVA batch execution API. The main difference is how the batch is triggered and configured in each CI system.

Common setup (required once)​

These steps are shared by all CI/CD integrations.

Prerequisites​

  • At least one test created in AIVA which has assigned label

Create an AIVA API key​

Generate an API key according to the How to run tests via API guide.

Store the key securely. You will use it as a secret or protected variable in your CI system.

GitHub Action​

To integrate with GitHub, use our AIVA GitHub Action.

Step 1. Add the API key as a repository secret​

Use the official GitHub guide to add the AIVA API key to your repository as a secret.

  • Name: "AIVA_API_KEY"
  • Value: your API key

Step 2. Create a Batch in AIVA​

Create a test batch in AIVA and copy its batch ID — the workflow runs an existing batch by this ID. See the Run test batches guide for how to create one.

Step 3. Create a workflow​

Create a file in the .github/workflows/ directory. You can give the file any name (or you can use aiva-integration.yml)

Add the workflow setup to the file

name: Run AIVA Test Batch

on:
  workflow_dispatch:
    inputs:
      BATCH_ID:
        description: "Id of the batch to run"
        type: string
        default: "123e4567-e89b-12d3-a456-426614174000"

jobs:
  aiva_batch_job:
    name: Run AIVA batch (example)
    runs-on: ubuntu-latest

    steps:
      - name: Start AIVA batch
        id: aiva
        uses: aiva-actions/run@main
        with:
          apiKey: ${{ secrets.AIVA_API_KEY }}
          batchId: ${{ inputs.BATCH_ID }}
    
      - name: Download Summary template
        id: template-download
        run: wget https://raw.githubusercontent.com/aiva-actions/run/refs/heads/main/summary-template.hbs
        if: always()

      - name: Generate CTRF summary
        id: summary
        uses: ctrf-io/[email protected]
        with:
          report-path: './batch-ctrf.json'
          template-path: 'summary-template.hbs'
          custom-report: true
        if: always()
More complex example
name: Run AIVA Test Batch in a more complex way

on:
  workflow_dispatch:
    inputs:
      BATCH_NAME:
        description: "(Optional) Name for the test batch"
        type: string
        default: "CI Batch"
      BATCH_ID:
        description: "Id of the batch to run"
        type: string
        default: "123e4567-e89b-12d3-a456-426614174000"
      POLL_PERIOD:
        description: "(Optional) Specifies how long to wait between API calls for the batch status."
        required: false
        type: number
        default: 10

jobs:
  aiva_batch_job:
    name: Run AIVA batch (example)
    runs-on: ubuntu-latest

    steps:
      - name: Start AIVA batch
        id: aiva
        uses: aiva-actions/run@main
        with:
          apiKey: ${{ secrets.AIVA_API_KEY }}
          batchId: ${{ inputs.BATCH_ID }}
          batchName: ${{ inputs.BATCH_NAME }}
          pollPeriodSeconds: ${{ inputs.POLL_PERIOD }}
    
      - name: Download Summary template
        id: template-download
        run: wget https://raw.githubusercontent.com/aiva-actions/run/refs/heads/main/summary-template.hbs
        if: always()

      - name: Generate CTRF summary
        id: summary
        uses: ctrf-io/[email protected]
        with:
          report-path: '${{ inputs.REPORT_FILEPATH }}'
          template-path: 'summary-template.hbs'
          custom-report: true
        if: always()

After adding the file, commit and push it to GitHub.

Step 3.1 Check formatting of workflow inputs​

Below is an example of workflow inputs so you can check the formatting.

Keep in mind that globalVariableOverrides and variableOverridesPerTest are different fields and cannot be mixed. See API Documentation for more details

apiKey: aiva_aaabbbcccddd... # NOTE: API should be put into Repository secrets, see Step 1.

batchName: CI Batch

batchId: 123e4567-e89b-12d3-a456-426614174000

pollPeriodSeconds: 10

Step 4. Run the workflow​

  1. Open your repository on GitHub
  2. Go to Actions
  3. Select Run AIVA test batch
  4. Click Run workflow
  5. Fill in the batch ID

Now run the workflow. You can watch the ongoing batch execution in AIVA Batch Runner.

Step 5. Outputs​

After the batch finishes running, GitHub will download CTRF results and upload them as an artifact. It will also write a test summary direclty to the workflow run.

GitLab CI​

GitLab integration uses the AIVA Runner executed inside a CI job.

Step 1. Add the API key as a hidden and protected variable​

In your GitLab project:

  1. Go to Settings -> CI/CD -> Variables
  2. Click Add variable in the Project Variables section

Add variable button

  1. Set:
    • Key: AIVA_API_KEY
    • Value: your API key
  2. Select Masked and hidden in Visibility section and check Protect Variable flag. For more context see CI/CD variable security.
  3. Click Add Variable button to save the new variable

Step 2. Create a GitLab pipeline​

Click Build -> Pipeline editor. This will open a page where you can edit your project's pipeline configuration right away. Add the following workflow setup to the file. Don't forget to replace the argument <BATCH_ID> with the ID of the batch you created in AIVA.

stages:
  - test

aiva-test:
  stage: test
  image: node
  script:
    - "npm install github:aiva-actions/runner"
    - npx runner --batch-id '<BATCH_ID>' -k $AIVA_API_KEY -b 'GitLab CI Batch' --result-format 'junit' -f ./junit-report.xml

  artifacts:
    when: always
    paths:
      - junit-report.xml
    reports:
      junit: junit-report.xml
More complex example
spec:
  inputs:
    aiva-batch-id:
      default: "123e4567-e89b-12d3-a456-426614174000"
    aiva-batch-name:
      default: "CI Batch"
---
stages:          # List of stages for jobs, and their order of execution
  - build
  - test

build:
  stage: build
  script:
    - echo "Your build process runs here."

aiva-test:
  stage: test
  image: node
  tags:
    - docker
  script:
    - "npm install github:aiva-actions/runner"
    - npx runner --batch-id '$[[ inputs.aiva-batch-id ]]' -k $AIVA_API_KEY -b '$[[ inputs.aiva-batch-name ]]' --result-format 'junit' -f junit-report.xml

  artifacts:
    when: always
    paths:
      - junit-report.xml
    reports:
      junit: junit-report.xml

After adding the file, click the Commit changes button. GitLab will execute the pipeline right away.

Step 2.1 Check formatting of runner arguments​

Below is an example of the runner arguments so you can check the formatting. See API Documentation for more details

apiKey: aiva_aaabbbcccddd... # NOTE: API should be put into Repository secrets, see Step 1.

batchName: CI Batch

batchId: 123e4567-e89b-12d3-a456-426614174000

pollPeriodSeconds: 10

Step 3. See results​

Since GitLab executed the pipeline right away, you can click the pipeline run ID above the pipeline configuration editor, and it will take you to the pipelines page. From there you can click the aiva-test job to see logs from the job.

After the job finishes and the JUnit report gets uploaded as artifact, a Tests tab will appear on the page where you can browse the results in a nice web format.