CI/CD Integration

CI mode in one flag

orbittest run --ci

CI mode switches OrbitTest to automation-friendly behavior: the browser is hidden, the local failure report server stays off, traces and screenshots are captured only on failure (by default), retried-then-passing tests are marked flaky, and machine-readable outputs are written alongside the HTML report. The console prints a summary built for pipeline logs:

OrbitTest CI Summary
--------------------
Status: PASSED
Total: 5
Passed: 5
Flaky: 0
Failed: 0
Skipped: 0
Duration: 28.42s
Report: reports/runs/<run-id>/report.html
Summary: reports/runs/<run-id>/summary.json
JUnit: reports/runs/<run-id>/junit.xml

Retries and flaky detection

In CI, one retry is a pragmatic default, networks blip, runners stall. The important part is honesty: a test that fails first and passes on retry does not become quietly green. It is recorded as flaky in the HTML report, report.json, and summary.json, so instability stays visible and fixable.

// Config-based
module.exports = {
  retries: 0,
  ci: { enabled: Boolean(process.env.CI), retries: 1 }
};

// Or per run
// orbittest run --ci --retries 2

Retries in CI only; locally you want to see every failure.

Failure control: fail-fast and max-failures

orbittest run --ci --fail-fast        # stop after the first failure
orbittest run --ci --max-failures 3  # stop after three failures

When a stop condition triggers, tests that never got scheduled are reported as skipped, so the summary still explains the whole run. Use fail-fast on pull-request pipelines where one failure means the PR needs work anyway; use plain full runs on main-branch pipelines where you want the complete picture.

Sharding: split the suite across jobs

orbittest run --ci --shard 1/4
orbittest run --ci --shard 2/4
orbittest run --ci --shard 3/4
orbittest run --ci --shard 4/4

Four jobs, each running a disjoint quarter of the registered tests.

--shard current/total splits registered tests by index. Every shard writes its own HTML, JSON, summary, and JUnit files, so each job uploads its own artifacts. You can also set the shard from the ORBITTEST_SHARD environment variable, convenient with CI matrix variables.

Workers and shards solve different problems: --workers parallelizes within one machine, --shard splits across machines. Big suites use both, for example two shards with two workers each.

GitHub Actions: annotations and a complete workflow

With --github-annotations, failed tests surface as error annotations and flaky tests as warning annotations, directly in the Actions log and the pull request UI, pointing at the failing source line.

name: e2e

on:
  pull_request:
  push:
    branches: [main]

jobs:
  orbittest:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        include:
          - shard: 1/2
            artifact: shard-1
          - shard: 2/2
            artifact: shard-2
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx orbittest run --ci --workers 2 --shard ${{ matrix.shard }} --github-annotations
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: orbittest-report-${{ matrix.artifact }}
          path: reports/

.github/workflows/e2e.yml — two shards, reports uploaded even on failure.

Note

The if: always() on the upload step is the line teams forget. Without it, failed runs, the ones whose evidence you need most, upload nothing.

Other CI systems

Nothing above is GitHub-specific except the annotations flag. On Jenkins, GitLab CI, or Azure DevOps: run npx orbittest run --ci, publish junit.xml through the platform’s test-report integration, and archive the reports/ directory as a build artifact. The standard CI environment variable, which all of these platforms set, is enough to activate the config-level CI defaults.

Frequently asked questions

Should the pipeline fail when a test is flaky?

By default a flaky-but-passed run exits successfully, with the flakiness recorded. If you want stricter gates, read summary.json in a follow-up step and fail when summary.flaky > 0.

How many workers per CI runner?

Start with 2 on standard hosted runners. Each worker is a Chrome instance; oversubscribing CPU slows everything and creates timeout flakes. Measure before raising.

How do shards stay balanced?

Tests are split by index, so shards have near-equal counts but not necessarily equal durations. If one shard is consistently slow, move slow tests or add a shard.

Related pages

• Reports & Diagnostics — HTML, JSON, and JUnit reports, trace timelines, Smart Report browser evidence, step mode, and report cleanup.
• Configuration — Every orbittest.config.js option explained: workers, retries, timeouts, browser display, CI behavior, and named environments.
• Environment Variables — ORBITTEST_CHROME_PATH, sharding, CI flags, ADB and device selection, and how to keep secrets out of test files.
• CLI Reference — Every OrbitTest command and flag: run, init, ui, forge, devices, doctor, and clean-reports, with recommended npm scripts.