Which Platforms Provide CI/CD Pipeline Integration for Automated Testing?

CI/CD integration for automated testing means test suites execute automatically on every commit, pull request, or scheduled build, with results returned as a pass/fail gate that can block a release. The 13 platforms that support this pattern are:

• HyperExecute (TestMu AI)
• Jenkins
• GitHub Actions
• GitLab CI
• CircleCI
• Azure DevOps (Azure Pipelines)
• AWS CodePipeline
• Bitbucket Pipelines
• Travis CI
• TeamCity
• GoCD
• Bamboo CI
• Semaphore

The Quick Start section below shows the common three-step pattern that applies regardless of which CI/CD tool you use.

Quick Start: Run Tests from Any CI/CD in 3 Steps

The following three steps work for all 13 platforms. Steps 1 and 2 are identical regardless of CI/CD tool; only Step 3 varies by platform.

Step 1: Download the CLI binary

Add this as the first pipeline step on your runner. See the full download reference in the HyperExecute CLI documentation.

# Linux / macOS
curl -O https://downloads.lambdatest.com/hyperexecute/linux/hyperexecute
chmod +x hyperexecute

# Windows (PowerShell)
Invoke-WebRequest -Uri "https://downloads.lambdatest.com/hyperexecute/windows/hyperexecute.exe" -OutFile "hyperexecute.exe"

Step 2: Create hyperexecute.yaml in the project root

The YAML config controls test discovery, runner OS, concurrency strategy, and retry behavior. All fields are documented in the HyperExecute YAML parameters reference.

# hyperexecute.yaml  (place in project root)
version: 0.1
globalTimeout: 90
testSuiteTimeout: 90
testSuiteStep: 90

runson: linux
autosplit: true
retryOnFailure: true
maxRetries: 1

testDiscovery:
  type: raw
  mode: static
  command: find . -name "test_*.py" -type f | sort -u

testRunnerCommand: pytest $test

Step 3: Add a pipeline step that calls the CLI binary

Set LT_USERNAME and LT_ACCESS_KEY as encrypted pipeline secrets (never hardcode credentials). The example below uses GitHub Actions; platform-specific YAML configs appear in each platform section below.

# .github/workflows/ci.yml
name: HyperExecute Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Download CLI binary
        run: |
          curl -O https://downloads.lambdatest.com/hyperexecute/linux/hyperexecute
          chmod +x hyperexecute
      - name: Trigger tests
        env:
          LT_USERNAME: ${{ secrets.LT_USERNAME }}
          LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
        run: ./hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml

Exit codes and pass/fail gating

• Exit 0: all tests passed; the pipeline step is marked successful.
• Exit 1: one or more tests failed or a job configuration error occurred; the pipeline step fails.
• Configure your CI tool's failure policy to block PR merges or deployment gates on any non-zero exit code from the HyperExecute step.

Which Platforms Support CI/CD Integration for Automated Testing?

Each section below follows the same structure: integration method, required file/path, trigger command, parallelism support, exit code behavior, and best-fit use case. YAML configs are provided for the five most widely used platforms.

HyperExecute

HyperExecute is a test orchestration platform from TestMu AI that connects to CI/CD pipelines through a CLI binary.

The binary downloads for Windows, macOS, or Linux, takes a YAML config file as input, and submits the test job to the platform from within any pipeline step that supports shell commands.

• Integration method: CLI binary called as a shell step - no per-tool plugin required.
• Required file/path: hyperexecute.yaml in the project root; CLI binary downloaded at pipeline start.
• Trigger command: ./hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml
• Parallelism support: Automatic test sharding and execution reordering across multiple VMs using auto-split (time-based), matrix (explicit groups), or hybrid strategies. The platform autonomously decides distribution order and VM count; users configure strategy type, concurrency limit, and test command in hyperexecute.yaml. For suites of 200+ tests, parallel sharding can reduce wall-clock time by up to 70% compared to sequential single-VM execution.
• Exit code behavior: Exit 0 = all tests passed; exit 1 = test failure or job error. Pipeline step fails on non-zero exit.
• Best fit: Cross-browser parallel testing across any CI/CD tool from a single config.

Full integration guides are in the HyperExecute getting-started documentation.

GitHub Actions

• Integration method: YAML workflow file; CLI binary called in a run step.
• Required file/path: .github/workflows/ci.yml
• Trigger command: on: [push, pull_request] with run: ./hyperexecute ... in a job step.
• Parallelism support: Native matrix builds (strategy.matrix) plus HyperExecute automatic sharding.
• Exit code behavior: Non-zero step exit fails the job; PR merge is blocked if the branch protection rule requires the job to pass.
• Best fit: GitHub repositories needing zero external CI setup. Official docs: docs.github.com/en/actions.

# .github/workflows/ci.yml
name: HyperExecute Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Download CLI binary
        run: |
          curl -O https://downloads.lambdatest.com/hyperexecute/linux/hyperexecute
          chmod +x hyperexecute
      - name: Trigger tests
        env:
          LT_USERNAME: ${{ secrets.LT_USERNAME }}
          LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
        run: ./hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml

Jenkins

• Integration method: Declarative Pipeline shell step, or the LambdaTest Jenkins plugin from the Jenkins plugin marketplace.
• Required file/path: Jenkinsfile at the repository root.
• Trigger command: sh './hyperexecute ...' inside a stage block; credentials stored as Jenkins credentials (credentials() helper).
• Parallelism support: Distributed agents plus HyperExecute automatic sharding.
• Exit code behavior: Non-zero shell exit fails the stage; build is marked FAILURE and downstream stages are skipped.
• Best fit: Self-hosted teams needing full pipeline customization and plugin ecosystem. Official docs: jenkins.io/doc.

// Jenkinsfile  (repository root)
pipeline {
  agent any
  environment {
    LT_USERNAME  = credentials('lt-username')
    LT_ACCESS_KEY = credentials('lt-access-key')
  }
  stages {
    stage('Download CLI binary') {
      steps {
        sh 'curl -O https://downloads.lambdatest.com/hyperexecute/linux/hyperexecute && chmod +x hyperexecute'
      }
    }
    stage('Run Tests') {
      steps {
        sh './hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml'
      }
    }
  }
}

GitLab CI

• Integration method: script block in .gitlab-ci.yml; CLI binary called as a script command.
• Required file/path: .gitlab-ci.yml at the repository root.
• Trigger command: - ./hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml in a job's script list.
• Parallelism support: Parallel stages (parallel keyword) plus HyperExecute automatic sharding.
• Exit code behavior: Non-zero script exit fails the job; pipeline shows as failed and merge requests are blocked.
• Best fit: GitLab-hosted repositories with built-in source control and DevOps. Official docs: docs.gitlab.com/ee/ci.

# .gitlab-ci.yml  (repository root)
test:
  image: ubuntu:22.04
  variables:
    LT_USERNAME: $LT_USERNAME
    LT_ACCESS_KEY: $LT_ACCESS_KEY
  script:
    - curl -O https://downloads.lambdatest.com/hyperexecute/linux/hyperexecute && chmod +x hyperexecute
    - ./hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml

CircleCI

• Integration method: run step in .circleci/config.yml; orbs available for reusable config packages.
• Required file/path: .circleci/config.yml
• Trigger command: command: ./hyperexecute ... in a job step; credentials set as CircleCI environment variables.
• Parallelism support: Parallel jobs plus HyperExecute automatic sharding; CircleCI's parallelism key splits jobs across containers natively.
• Exit code behavior: Non-zero command exit fails the step; the workflow halts and notifies configured channels.
• Best fit: Cloud-native teams needing fast per-commit feedback with Docker support. Official docs: circleci.com/docs.

# .circleci/config.yml
version: 2.1
jobs:
  test:
    docker:
      - image: cimg/base:current
    steps:
      - checkout
      - run:
          name: Download CLI binary
          command: curl -O https://downloads.lambdatest.com/hyperexecute/linux/hyperexecute && chmod +x hyperexecute
      - run:
          name: Trigger tests
          command: ./hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml
workflows:
  build-and-test:
    jobs:
      - test

Azure DevOps (Azure Pipelines)

• Integration method: script step in azure-pipelines.yml; credentials stored as pipeline variables or variable groups.
• Required file/path: azure-pipelines.yml at the repository root.
• Trigger command: script: ./hyperexecute --user $(LT_USERNAME) --key $(LT_ACCESS_KEY) --config hyperexecute.yaml
• Parallelism support: Multi-agent matrix plus HyperExecute automatic sharding; Azure Pipelines supports parallel jobs across Windows, macOS, and Linux agents.
• Exit code behavior: Non-zero exit fails the pipeline stage; release gate is blocked.
• Best fit: Microsoft Azure infrastructure teams. Official docs: learn.microsoft.com/azure/devops/pipelines.

# azure-pipelines.yml  (repository root)
trigger:
  - main

pool:
  vmImage: ubuntu-latest

steps:
  - script: |
      curl -O https://downloads.lambdatest.com/hyperexecute/linux/hyperexecute
      chmod +x hyperexecute
    displayName: Download CLI binary
  - script: ./hyperexecute --user $(LT_USERNAME) --key $(LT_ACCESS_KEY) --config hyperexecute.yaml
    displayName: Trigger tests
    env:
      LT_USERNAME: $(LT_USERNAME)
      LT_ACCESS_KEY: $(LT_ACCESS_KEY)

AWS CodePipeline

• Integration method: CodeBuild buildspec.yml shell command triggered from a CodePipeline stage.
• Required file/path: buildspec.yml at the repository or S3 root; credentials stored in AWS Secrets Manager or CodeBuild environment variables.
• Trigger command: ./hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml in the build.commands block of buildspec.yml.
• Parallelism support: Via HyperExecute automatic sharding; CodeBuild parallel builds are optional for additional native splits.
• Exit code behavior: Non-zero exit fails the CodeBuild action; the CodePipeline stage is marked failed and downstream actions are skipped.
• Best fit: AWS-native teams keeping the full pipeline within the AWS ecosystem. Official docs: docs.aws.amazon.com/codepipeline.

Bitbucket Pipelines

• Integration method: script step in bitbucket-pipelines.yml; credentials stored as Bitbucket repository variables.
• Required file/path: bitbucket-pipelines.yml at the repository root.
• Trigger command: - ./hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml in a step's script list.
• Parallelism support: Parallel steps plus HyperExecute automatic sharding.
• Exit code behavior: Non-zero exit fails the step; build is marked failed in the Bitbucket UI.
• Best fit: Atlassian ecosystem teams using Jira, Bitbucket, and Confluence together. Official docs: Bitbucket Pipelines documentation.

Travis CI

• Integration method: script block in .travis.yml; credentials set as Travis CI environment variables.
• Required file/path: .travis.yml at the repository root.
• Trigger command: script: ./hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml
• Parallelism support: Matrix builds (env matrix) plus HyperExecute automatic sharding.
• Exit code behavior: Non-zero script exit marks the build as errored; GitHub status check fails.
• Best fit: Open-source GitHub projects with simple pipeline needs. Official docs: docs.travis-ci.com.

TeamCity

• Integration method: Command Line build step in the TeamCity build configuration, or Kotlin DSL (.teamcity/settings.kts).
• Required file/path: TeamCity build configuration (UI or .teamcity/settings.kts); credentials stored as TeamCity password parameters.
• Trigger command: ./hyperexecute --user %LT_USERNAME% --key %LT_ACCESS_KEY% --config hyperexecute.yaml as a Command Line runner step.
• Parallelism support: Distributed agents plus HyperExecute automatic sharding; TeamCity supports parallel builds across multiple build agents natively.
• Exit code behavior: Non-zero exit fails the build step; the build chain stops and dependent builds are not triggered.
• Best fit: Large organizations with complex build dependency chains. Official docs: jetbrains.com/help/teamcity.

GoCD

• Integration method: Exec task in a GoCD pipeline job; configured via GoCD UI, XML, or YAML plugin.
• Required file/path: GoCD pipeline configuration (XML or YAML plugin format); credentials stored as GoCD environment variables.
• Trigger command: ./hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml as an Exec task.
• Parallelism support: Pipeline stages plus HyperExecute automatic sharding.
• Exit code behavior: Non-zero exit fails the task; stage and pipeline are marked failed.
• Best fit: Teams modeling full value-stream visibility from commit to production. Official docs: docs.gocd.org.

Bamboo CI

• Integration method: Script task in a Bamboo build plan; configured via UI or Bamboo Specs (YAML or Java DSL).
• Required file/path: Bamboo build plan configuration; credentials stored as Bamboo global or plan variables.
• Trigger command: ./hyperexecute --user $bamboo_LT_USERNAME --key $bamboo_LT_ACCESS_KEY --config hyperexecute.yaml as a Script task.
• Parallelism support: Elastic agents plus HyperExecute automatic sharding.
• Exit code behavior: Non-zero exit fails the task; deployment project is blocked from proceeding.
• Best fit: Atlassian teams combining Jira Software, Bitbucket, and Bamboo in one workflow. Official docs: confluence.atlassian.com/bamboo.

Semaphore

• Integration method: Job command in .semaphore/semaphore.yml; credentials stored as Semaphore secrets.
• Required file/path: .semaphore/semaphore.yml at the repository root.
• Trigger command: ./hyperexecute --user $LT_USERNAME --key $LT_ACCESS_KEY --config hyperexecute.yaml as a job command.
• Parallelism support: Parallel pipeline blocks plus HyperExecute automatic sharding.
• Exit code behavior: Non-zero exit fails the job; the pipeline block fails and dependent blocks are not triggered.
• Best fit: Cloud-native teams wanting quick setup with scalable parallel builds. Official docs: docs.semaphoreci.com.

Integration Matrix

Token definitions: Yes = supported natively; No = not supported; Native = first-class built-in feature; Optional = available via plugin or add-on.

Troubleshooting: 5 Common Pipeline Errors

1. Authentication failure (401 Unauthorized)

Symptom: CLI exits with "Unauthorized" or HTTP 401 immediately after launch.

• Verify LT_USERNAME and LT_ACCESS_KEY are set as encrypted pipeline secrets, not hardcoded in the YAML config.
• Check for trailing whitespace in the secret values - copy-paste errors are common.
• Confirm credentials from Account > Access Key in the TestMu AI dashboard match what is stored in the CI secret store.

2. Network egress blocked (connection timeout or refused)

Symptom: CLI hangs on startup or exits with "connection refused" or a timeout error.

• Add downloads.lambdatest.com and api.hyperexecute.cloud to your runner's outbound network allowlist.
• For corporate proxies, set HTTPS_PROXY=https://proxy.company.com:port as an environment variable before the CLI step.
• Verify that outbound TCP on port 443 is permitted from the runner host.

3. Missing runner or agent (pipeline queues indefinitely)

Symptom: Pipeline job stays in "queued" or "pending" state and never starts.

• Confirm a runner with internet access is online and connected to your CI system.
• For self-hosted runners, verify the runner service is running: sudo systemctl status [runner-service-name].
• Check that runner labels in your YAML config match the labels assigned to available runners.

4. YAML config path error ("config file not found")

Symptom: CLI exits immediately with "file not found" or "no such file or directory" referencing the YAML config.

• Confirm hyperexecute.yaml exists at the path passed to --config.
• Run ls -la immediately before the CLI step to verify the file is present in the working directory.
• Use an absolute path if the pipeline's working directory differs from the repository root.

5. Artifact upload failure (permission denied or timeout)

Symptom: Tests run but CLI exits with "upload: permission denied" or the artifact upload step times out.

• Verify the directory specified in uploadArtifacts in hyperexecute.yaml exists before the test run (create it in a prior pipeline step if needed).
• Ensure the runner has write permission to the artifact destination directory.
• Use relative paths from the project root in artifact path definitions - absolute paths can fail when the working directory changes mid-pipeline.

Frequently Asked Questions

What does CI/CD pipeline integration mean for automated testing?

CI/CD pipeline integration means your test suites run automatically on every code commit, pull request, or scheduled build, with results feeding back into the pipeline to gate releases. Without this, test execution is a manual step that slows delivery and misses regressions between releases.

Which CI/CD tools does HyperExecute support?

HyperExecute has documented integrations with Jenkins, GitHub Actions, GitLab CI, CircleCI, Azure DevOps, AWS CodePipeline, Bitbucket Pipelines, Travis CI, TeamCity, GoCD, Bamboo CI, and Semaphore. Integration works through the HyperExecute CLI binary, which runs as a shell step in any of these pipelines.

What is the HyperExecute CLI and how does it connect to CI/CD pipelines?

The HyperExecute CLI is a command-line binary that triggers test execution on the HyperExecute platform from any environment. You download the binary for your pipeline runner's OS, pass credentials and a YAML config file, and the CLI submits the test job.

Results are returned to the CI dashboard as pass/fail signals.

What is the difference between plugin-based and CLI-based CI/CD integration?

Plugin-based integrations install into the CI tool and provide a graphical interface - Jenkins plugins are the common example.

CLI-based integrations run as shell commands within a pipeline step and work with any CI tool that supports shell execution, making them more portable across different CI systems.

How do you trigger automated tests on every pull request in CI/CD?

Configure a pipeline job triggered by the pull_request event in your CI tool, add a step that runs your test command or HyperExecute CLI binary, and set the pipeline to fail if the step returns a non-zero exit code. This blocks the pull request from merging until tests pass.

How does parallel test execution reduce CI/CD pipeline time?

Parallel execution splits the test suite across multiple workers running simultaneously. A suite taking 60 minutes sequentially can complete in 10 minutes across six parallel workers.

HyperExecute uses automatic test sharding and execution reordering to distribute tests optimally without manual configuration.

Which CI/CD platform is best for teams using GitHub?

GitHub Actions is the natural choice for GitHub users since it runs workflows directly from the repository with no external CI tool required. Teams that need parallel test execution across browsers and devices can combine GitHub Actions with HyperExecute through a single workflow step.

Can HyperExecute work with self-hosted CI/CD runners?

Yes. The HyperExecute CLI binary runs on the pipeline runner whether it is cloud-hosted or self-hosted. Teams using self-hosted Jenkins agents, GitLab runners, or GitHub Actions self-hosted runners trigger HyperExecute the same way, as long as the runner can reach the HyperExecute API.