Shiplight CI Runners

View as Markdown

Run Shiplight E2E tests on Shiplight-hosted GitHub Actions runners — ephemeral VMs with Chromium and Playwright pre-installed. Test results are automatically uploaded to Shiplight Cloud for trend tracking, flaky-test detection, and team visibility.

Let your agent wire this up: /shiplight ci

The fastest way to author both the CI test workflow and the auto-triage pipeline is to run /shiplight ci in your coding agent. It picks the runner option, generates the workflow files, and wires the report-upload and triage steps for you. Use this page to understand what it produces and to configure it by hand when needed.

Prerequisites

1. Install the Shiplight GitHub App on your repository (or organization). The app requires Actions and Self-hosted runners permissions.
2. Enable Shiplight Runners — an org owner toggles the switch in Organization Settings.

Minimal Workflow Example

Create .github/workflows/e2e.yml:

name: E2E Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e:
    runs-on: shiplight-small
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v5

      - name: Install dependencies
        working-directory: tests/e2e
        run: npm install

      - name: Run E2E tests
        working-directory: tests/e2e
        run: npx shiplight test

      - name: Upload results to Shiplight
        if: always()
        working-directory: tests/e2e
        run: npx shiplight report

That's it. The sections below explain each part in detail.

Runner Sizes

Use runs-on to select the VM size for your job:

runs-on: shiplight-small

When the workflow is triggered, Shiplight provisions an ephemeral VM, registers it as a GitHub self-hosted runner, and destroys it after the job completes.

Label	vCPUs	Memory
shiplight-small	4	16 GB
shiplight-medium	8	32 GB
shiplight-large	16	64 GB
shiplight-xlarge	32	128 GB

No browser install needed

The runner image ships with Chromium and Playwright pre-installed. Do not run npx playwright install chromium in your workflow — it's already there.

What the Runner Provides

Shiplight runners come pre-configured with everything needed to run and report E2E tests:

• Action cache — when a cached locator self-heals during a CI run, the updated locator is persisted back to Shiplight Cloud so future runs replay at full speed without manual intervention.
• Chromium + Playwright — pre-installed, no npx playwright install step needed.
• Automatic result reporting — shiplight report works out of the box with no tokens or API URLs to configure. Credentials are provisioned per run and revoked automatically when the run completes.
• LLM access — if your tests use AI-powered actions (natural language steps, self-healing locators), the runner provides LLM credentials automatically. No API keys needed in your workflow.

Upload results

- name: Upload results to Shiplight
  if: always()
  working-directory: tests/e2e
  run: npx shiplight report

Uploads test results — including per-step screenshots, videos, and traces — to Shiplight Cloud. Results appear in the Test Results section, linked to your organization.

WARNING

Always use if: always() so results are uploaded even when tests fail. Without it, a red test run produces no cloud report.

On Shiplight runners, no additional configuration is needed — credentials are provided automatically.

On non-Shiplight runners

shiplight report also works on any other CI provider (GitHub-hosted runners, GitLab, CircleCI, etc.). You need to set two environment variables: SHIPLIGHT_REPORT_TO_CLOUD=1 to enable the upload (Shiplight runners set this automatically; other runners don't), and SHIPLIGHT_API_TOKEN to an org API token from nova.shiplight.ai/api-tokens:

- name: Upload results to Shiplight
  if: always()
  working-directory: tests/e2e
  env:
    SHIPLIGHT_REPORT_TO_CLOUD: "1"
    SHIPLIGHT_API_TOKEN: ${{ secrets.SHIPLIGHT_API_TOKEN }}
  run: npx shiplight report

Store the token as a repository or organization secret in your CI provider, then reference it as shown above.

Sharded runs (GHA matrix)

When using Playwright sharding with a GitHub Actions matrix, each shard produces its own report. Use shiplight report --merge to combine them into a single upload so the cloud dashboard shows one unified run instead of N separate ones.

Auto-triage CI failures (optional)

When an E2E workflow goes red, Shiplight can have an AI agent diagnose the failure and — for fixable spec issues — repair it automatically. The pipeline lives in the reusable workflow ShiplightAI/ci-triage. On a failed run it reads the run logs and the uploaded report artifacts (screenshots, traces), posts a diagnosis to Slack, and for failures it classifies as fixable spec issues it applies the fix, re-runs the test, and opens a PR. It never auto-merges.

Set this up only once a test workflow (from the examples above) exists — triage triggers off that workflow's completion. Wiring is two steps.

Step 1 — upload the report artifact from the test workflow

Triage reads failure evidence from a GitHub artifact, which is separate from the cloud upload. Add this step to your e2e.yml, after the test step:

- name: Upload test report (for triage)
  if: ${{ !cancelled() }}
  uses: ShiplightAI/ci-triage/upload-report@v1
  # sharded/matrix jobs: give each shard a unique name
  # with:
  #   name: test-report-shard-${{ matrix.shardIndex }}
  #   retention-days: "1"

The helper bakes in the shiplight-report/ path and drops the heavy Playwright traces (*.zip) and videos (*.webm) the agent never reads (~80% of the artifact size). Keep your npx shiplight report step too — the cloud report still gets full traces and videos for humans.

Step 2 — add the caller workflow

The workflow_run trigger and the per-repo credential mapping must live in your repo — a workflow_run trigger is illegal inside a reusable workflow, and secret names differ per repo. Everything else lives in the reusable workflow. Create .github/workflows/ci-failure-triage.yml:

name: CI Failure Triage

on:
  workflow_run:
    workflows: [E2E Tests] # exact `name:` of each test workflow to watch
    types: [completed]

jobs:
  triage:
    uses: ShiplightAI/ci-triage/.github/workflows/triage.yml@v1 # pin to an immutable tag, not @main
    permissions:
      contents: write
      pull-requests: write
      actions: read
    with:
      triage-runner: ubuntu-latest # read-only diagnosis job
      autofix-runner: shiplight-medium # re-runs tests, so needs browsers/network
      node-version: "20"
      allowed-paths: "tests templates" # top-level dirs the autofix agent may edit (hard guard)
      slack-channel: ${{ vars.SLACK_CHANNEL_ID || '' }} # empty disables Slack
    secrets:
      claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
      openai_api_key: ${{ secrets.OPENAI_API_KEY }} # Codex fallback when Claude is unavailable
      autofix_github_token: ${{ secrets.AUTOFIX_GITHUB_TOKEN }} # PAT/App token to open the PR; falls back to GITHUB_TOKEN
      slack_bot_token: ${{ secrets.SLACK_BOT_TOKEN }}
      # Per-repo credential mapping → generic env for the autofix re-run.
      # One KEY=VALUE per line; values must be single-line.
      extra_env: |
        BASE_URL=${{ vars.BASE_URL || 'https://example.com' }}
        MY_TEST_USER_PASSWORD=${{ secrets.MY_TEST_USER_PASSWORD }}
        MY_TEST_USER_2FA_SECRET=${{ secrets.MY_TEST_USER_2FA_SECRET }}

Notes:

• workflows: must list the exact name: of each test workflow to watch. Never list the triage workflow itself there.
• extra_env maps your repo's secret names onto the generic env the autofix job uses, so npx shiplight test and the MCP browser can authenticate during the re-run. Mirror the env: block from your test workflow.
• Provide at least one model credential (claude_code_oauth_token or anthropic_api_key); openai_api_key enables the Codex fallback. autofix_github_token and slack_bot_token are optional.
• autofix-runner re-runs the failing test, so it needs browsers and network — use a Shiplight runner, or install Chromium on a stock runner the same way your test workflow does.
• These examples assume the Shiplight project lives at the repository root. If it lives in a subdirectory, adjust the report path and add working-directory to the relevant steps.

WARNING

The triage job runs privileged — contents: write plus live credentials for the autofix re-run. Pin uses: to an immutable tag (@v1.x.y), never a moving branch like @main.

Troubleshooting

Job stays queued

• Confirm Shiplight Runners are enabled in Org Settings.
• Verify the Shiplight GitHub App is installed on the repository with the required permissions.
• Check whether your organization's runner concurrency limit has been reached. Contact your Shiplight admin to raise the limit if needed.

Test results not appearing in the dashboard

• Make sure the shiplight report step has if: always() so it runs after failures.
• Check the step logs for errors.

Chromium is downloading during CI

The runner has Chromium pre-installed. Do not add npx playwright install chromium to your workflow. If Playwright still downloads a browser, check that your package.json does not override the Playwright version bundled with shiplightai.

Related

• CLI Reference — full shiplight test, shiplight report, and other command documentation
• GitHub Actions Integration — trigger Shiplight Cloud test suites from GitHub Actions (a different integration path)
• Local Testing — project structure, authentication, and configuration