CI/CD

View as Markdown

Run your Shiplight E2E tests in CI on every push or pull request, and upload the results to Shiplight Cloud for trend tracking, flaky-test detection, and team visibility.

Not sure whether CI should run on your own provider's machines or on Shiplight-hosted runners? Start with Where tests run.

How it fits together

Three separate pieces, and it pays to keep them distinct:

• Your Shiplight test project: tests/, specs/, and a package.json that includes the shiplightai CLI. Scaffolded by /shiplight init; it must exist first.
• A CI config file: tells your CI provider how to install dependencies, run shiplight test, and upload with shiplight report. One file per provider, at your repository root.
• Your CI provider: GitHub Actions, GitLab, etc. It detects the config file and runs it automatically on the trigger you set (push / PR / MR).

The config file does not create tests; it runs an existing project. On every provider the job boils down to the same four commands:

npm install                       # installs the shiplight CLI (shiplightai) + project deps
npx playwright install chromium   # the browser the tests drive
npx shiplight test                # run the tests
npx shiplight report              # upload results (run even when tests fail)

Let your agent set it up

Once tests exist, your agent can wire them into CI for you:

/shiplight ci

The agent looks at your repo, picks the right runner setup, writes the workflow, and tells you which secrets to add. You get a working workflow file in your repo (for GitHub Actions, .github/workflows/e2e.yml) that:

• Runs the suite on every push to main and every pull request
• Uploads results to Shiplight Cloud even when tests fail, so a red run still produces a report with screenshots, videos, and traces

The one manual step: create an API token at app.shiplight.ai/api-tokens and store it as the SHIPLIGHT_API_TOKEN secret.

For providers other than GitHub Actions, copy the template from the matching page under Other providers.

Where the job runs

The commands above are the same wherever the job executes. What differs is the machine:

• Your CI provider's runners: GitHub's ubuntu-latest, GitLab shared runners, and so on. You install the browser and supply SHIPLIGHT_API_TOKEN or your own LLM credentials; a Shiplight token is still required for report uploads and the shared Cloud action cache.
• Shiplight CI runners: ephemeral Shiplight-hosted VMs with Chromium, Firefox, and WebKit preinstalled and credentials injected per run. GitHub Actions only, and enabled by changing a single runs-on line.

Where tests run compares the two (and local runs) with the cost implications of each.

GitHub is first-class

GitHub Actions is the primary integration. Only there do you get Shiplight-hosted ephemeral runners (Chromium pre-installed and credentials auto-provisioned) and the failure auto-triage pipeline. Runtime self-healing is part of the shiplight CLI on every provider, and the shared Shiplight Cloud action cache works in any CI environment configured with CI=true and SHIPLIGHT_API_TOKEN. If you're on GitHub, start there.

Auto-triage failures

/shiplight ci can also wire a second workflow that watches the first. When a test run goes red, an AI agent reads the run logs and the uploaded failure evidence (screenshots and traces) and:

1. Posts a diagnosis to Slack, so the team knows what broke without opening CI
2. For fixable test issues, applies the repair itself: the CI equivalent of running fix locally. It re-runs the test to confirm, then opens a PR with the fix

It never auto-merges, and the autofix agent can only touch the paths you allow (your test directories). Ask for it once the test workflow exists:

/shiplight ci, add the failure triage pipeline to the E2E workflow

Triage builds on GitHub's workflow_run trigger, so it's GitHub Actions only. Full wiring and configuration: Auto-triage CI failures.

Reading results

Each run lands in Shiplight Cloud with its artifacts. Across runs you get trends, flaky-test detection, and failure attribution. Ask for them right from your agent:

/shiplight cloud, show failing tests from the last run

Other providers

The shiplight CLI runs on any pipeline that has Node.js 22 or newer. Each page below gives a copy-paste template. Shiplight-hosted runners, automatically provisioned credentials, and auto-triage are GitHub Actions only; everywhere else you supply the API token and install Chromium yourself.

AI-powered steps, including runtime self-healing, work on every provider when the job has SHIPLIGHT_API_TOKEN or supported bring-your-own-model credentials. With CI=true and a Shiplight token that has action-cache:read and action-cache:write, healed locators from passing tests are stored in the shared Shiplight Cloud action cache and can be reused by later runs.

• GitLab CI
• Depot CI
• CircleCI
• Jenkins
• Travis CI
• Buildkite
• Azure DevOps
• Custom setups: the generic recipe for any other provider

Slow downloads in CI

On hosted runners with restricted or high-latency network access, installing dependencies or Playwright browsers may take significantly longer than expected.

Use a trusted regional mirror, proxy, or prebuilt container image provided by your organization or CI provider. A prebuilt image should include Node.js 22 or newer, the required browser dependencies, and a Playwright browser version compatible with the project.

Related

• Where tests run: local vs Shiplight runners vs your own CI, and what each costs
• CLI Reference: full shiplight test / shiplight report documentation
• Run Locally: project structure, authentication, and configuration