Tests Buildkite Plugin

The Buildkite plugin for automatically parallelising and running your test suites on Buildkite. It splits your tests across parallel jobs so they finish faster, and handles all the setup for you. The splitting and running is done by bktec, the client this plugin installs and configures for you.

Behind the scenes, the plugin downloads bktec, requests an OIDC token, ensures your test suite exists, and exports the environment variables bktec expects, so you do not have to install bktec, set up authentication, or create your test suite by hand. It works with every test runner that bktec supports, including RSpec, Jest, pytest, and Go test.

How it works

sequenceDiagram
    participant Pipeline as Pipeline YAML
    participant Agent as Buildkite Agent
    participant Plugin as Tests Plugin
    participant OIDC as Buildkite OIDC
    participant API as Test Engine API
    participant GH as GitHub Releases
    participant bktec
    participant Runner as Test Runner

    Pipeline->>Agent: step with tests plugin and bktec run command

    Note over Agent,Plugin: environment hook
    Plugin->>Plugin: map plugin options to BUILDKITE_TEST_ENGINE_* env vars

    Note over Agent,Plugin: pre-command hook
    Plugin->>OIDC: request suite-scoped OIDC token
    OIDC-->>Plugin: token
    Plugin->>API: GET /suites/{slug}
    alt suite does not exist (404 or 401)
        Plugin->>OIDC: request org-scoped OIDC token
        OIDC-->>Plugin: org token
        Plugin->>API: POST /suites/create_from_pipeline
        API-->>Plugin: suite created
    end
    Plugin->>GH: download bktec binary (if install-client: true)
    GH-->>Plugin: verified binary → added to PATH

    Note over Agent,bktec: command step
    Agent->>bktec: bktec run
    bktec->>API: fetch test plan for this node
    API-->>bktec: assigned test files
    bktec->>Runner: run assigned tests
    Runner-->>bktec: results
    bktec->>API: upload results

Example

steps:
  - label: "RSpec"
    command: bktec run
    plugins:
      - tests#v1.0.0:
          test-runner: rspec
          result-path: tmp/rspec-result.json
    parallelism: 2

Each step must invoke bktec run (or bktec plan) in its command — the plugin sets up the environment but does not run the test command for you. The plugin downloads bktec by default; set install-client: false if bktec is already installed on the agent.

If the suite-slug attribute is not set, the plugin uses the pipeline slug as the test suite slug. If the test suite does not exist yet, the plugin creates it from the pipeline on the first run.

Examples by runner

Jest

steps:
  - label: "Jest"
    command: bktec run
    plugins:
      - tests#v1.0.0:
          test-runner: jest
          result-path: jest-results.json
    parallelism: 4

pytest

steps:
  - label: "Run tests"
    command: bktec run
    parallelism: 5
    plugins:
      - tests#v1.0.0:
          test-runner: pytest

Go test

steps:
  - label: "Go test"
    command: bktec run
    plugins:
      - tests#v1.0.0:
          test-runner: gotest
          result-path: gotest-results.xml
    parallelism: 4

Dynamic parallelism

To let bktec choose the number of partitions for a target run time, run bktec plan in a planning step with the --pipeline-upload flag, pointing at a pipeline template you supply. bktec plan uploads a follow-up step that runs the test plan. --pipeline-upload is passed directly to bktec in command: because it is not exposed as a plugin option (see Unsupported bktec flags).

steps:
  - label: "Plan RSpec"
    key: rspec-plan
    command: bktec plan --pipeline-upload .buildkite/rspec-template.yml
    plugins:
      - tests#v1.0.0:
          test-runner: rspec
          result-path: tmp/rspec-result.json
          max-parallelism: 10
          target-time: 4m30s

The pipeline template at .buildkite/rspec-template.yml contains the run step:

steps:
  - label: "RSpec"
    key: rspec-run
    depends_on: rspec-plan
    command: bktec run --plan-identifier ${BUILDKITE_TEST_ENGINE_PLAN_IDENTIFIER}
    parallelism: ${BUILDKITE_TEST_ENGINE_PARALLELISM}
    plugins:
      - tests#v1.0.0:
          test-runner: rspec
          result-path: tmp/rspec-result.json

Docker

When the test command runs inside a Docker container, the plugin downloads bktec on the host (the default) and sets BUILDKITE_TEST_ENGINE_CLIENT_PATH that you can mount into the container. Set propagate-environment: true on the Docker plugin so the container picks up the BUILDKITE_TEST_ENGINE_* variables that are required to run bktec. Set client-os and client-arch when the host operating system or architecture differs from the container.

steps:
  - label: "Jest"
    command: bktec run
    plugins:
      - tests#v1.0.0:
          test-runner: jest
          client-os: linux
          result-path: jest-results.json
      - docker#v5.13.0:
          image: node:24-slim
          expand-volume-vars: true
          volumes:
            - "$$BUILDKITE_TEST_ENGINE_CLIENT_PATH:/usr/local/bin/bktec"
          propagate-environment: true

Configuration

bktec options

These options map directly to bktec flags or environment variables. For the full reference, see the Test Engine Client repository.

`test-runner` (required, string)

Any runner supported by bktec, for example rspec, jest, pytest, or gotest.

`test-cmd` (required for `custom` and `pytest-pants` runners, string)

Command to run tests.

`result-path` (string)

Path to the test runner's output file. Required for all runners except cypress, pytest, pytest-pants, and custom.

`suite-slug` (optional, string)

Test suite slug. Defaults to the pipeline slug.

`test-file-pattern` (required for `custom` runner, string)

Glob pattern for test files.

`test-file-exclude-pattern` (optional, string)

Exclude test files matching this pattern.

`tag-filters` (optional, string)

Filter tests by tag. Currently only supported by pytest.

`files` (optional, string)

Path to a file that lists test files to run, one per line.

`retry-count` (optional, integer)

Number of times to retry failing tests.

`retry-cmd` (optional, string)

Command to run when retrying failed tests. Defaults to test-cmd.

`disable-retry-for-muted-test` (optional, boolean)

Disable retry for muted tests.

`split-by-example` (optional, boolean)

Enable example-level splitting. Not supported by every runner.

`location-prefix` (optional, string)

Prefix to prepend to test file paths when requesting a test plan.

`max-parallelism` (optional, integer)

Maximum parallelism for dynamic test plans. Used with bktec plan.

`target-time` (optional, string)

Desired target time for the test suite, for example 4m30s. Must be used with max-parallelism.

`plan-identifier` (optional, string)

Test plan identifier created by bktec plan. Set automatically in dynamic mode; can be set manually for static mode.

`fail-on-no-tests` (optional, boolean)

Exit with an error if no tests are assigned to this node.

`upload-results` (optional, boolean)

Upload test results to your test suite. Default: true. Set to false if you use the test collectors for richer data collection, or if you want to handle the upload yourself.

Requires bktec 2.7.0 or later.

`tags` (optional, array of strings)

Tags to attach to the uploaded test results. Each tag is a key=value string.

plugins:
  - tests#v1.0.0:
      test-runner: rspec
      tags:
        - "language.version=3.3"
        - "os=linux"

`debug-enabled` (optional, boolean)

Enable verbose output.

Plugin options

These options control plugin behavior and are not passed to bktec.

`install-client` (optional, boolean)

Download the bktec binary, add it to the PATH, and set BUILDKITE_TEST_ENGINE_CLIENT_PATH. Default: true. Set to false if bktec is already installed on the agent.

`client-os` (optional, string)

Target OS for the downloaded binary, for example linux or darwin. Defaults to the host OS. Useful when the host and a Docker container run different operating systems.

`client-arch` (optional, string)

Target architecture for the downloaded binary, for example amd64 or arm64. Defaults to the host architecture.

`client-version` (optional, string)

bktec version to download, for example 2.4.0. Defaults to the latest release.

`oidc-lifetime` (optional, integer)

Lifetime in seconds for the OIDC token. Default: 300.

Unsupported bktec flags

A small number of bktec flags cannot be set through environment variables, so the plugin intentionally does not expose them as options. To use any of these, pass them directly to bktec in your step's command:.

Flag	Command	Notes
`--selection-param`	`run`, `plan`	Preview selection
`--metadata`	`run`, `plan`	Preview selection
`--json`	`plan`	Print the plan as JSON to stdout
`--pipeline-upload`	`plan`	Upload a follow-up pipeline step that runs the plan
`--output`	`backfill-commit-metadata`	Write tarball locally
`--upload`	`backfill-commit-metadata`	Upload an existing tarball
`--version`	Global	No environment variable needed

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/buildkite-plugins/tests-buildkite-plugin.

License

MIT (see LICENSE.txt).

Name		Name	Last commit message	Last commit date
Latest commit History 35 Commits
.buildkite		.buildkite
.github		.github
hooks		hooks
lib		lib
tests		tests
.gitignore		.gitignore
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
LICENSE.txt		LICENSE.txt
Makefile		Makefile
README.md		README.md
docker-compose.yml		docker-compose.yml
plugin.yml		plugin.yml

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Tests Buildkite Plugin

How it works

Example

Examples by runner

Jest

pytest

Go test

Dynamic parallelism

Docker

Configuration

bktec options

test-runner (required, string)

test-cmd (required for custom and pytest-pants runners, string)

result-path (string)

suite-slug (optional, string)

test-file-pattern (required for custom runner, string)

test-file-exclude-pattern (optional, string)

tag-filters (optional, string)

files (optional, string)

retry-count (optional, integer)

retry-cmd (optional, string)

disable-retry-for-muted-test (optional, boolean)

split-by-example (optional, boolean)

location-prefix (optional, string)

max-parallelism (optional, integer)

target-time (optional, string)

plan-identifier (optional, string)

fail-on-no-tests (optional, boolean)

upload-results (optional, boolean)

tags (optional, array of strings)

debug-enabled (optional, boolean)

Plugin options

install-client (optional, boolean)

client-os (optional, string)

client-arch (optional, string)

client-version (optional, string)

oidc-lifetime (optional, integer)