GitHub Actions

View as Markdown

Run Shiplight E2E tests on Shiplight-hosted GitHub Actions runners — ephemeral VMs with Chromium and Playwright pre-installed. Test results are automatically uploaded to Shiplight Cloud for trend tracking, flaky-test detection, and team visibility.

Let your agent wire this up: /shiplight ci

The fastest way to author both the CI test workflow and the auto-triage pipeline is to run /shiplight ci in your coding agent. It picks the runner option, generates the workflow files, and wires the report-upload and triage steps for you. Use this page to understand what it produces and to configure it by hand when needed.

Prerequisites

1. Install the Shiplight GitHub App from Settings → Integrations at app.shiplight.ai/settings/integrations, and make sure the repository is included in the App's repository selection.
2. Enable Shiplight Runners — an org owner turns on the toggle at app.shiplight.ai/runners.

The repository must be owned by a GitHub organization

Runner registration tokens are minted at the organization level, so repositories under a personal GitHub account cannot use Shiplight runners — jobs stay queued and the dispatch fails with Token mint failed.

See CI Runners for runner sizes, pool limits, and dispatch troubleshooting.

Minimal Workflow Example

In your existing E2E workflow, switch the runner and add Shiplight's test and report commands:

jobs:
  e2e:
    runs-on: ubuntu-latest
    runs-on: shiplight-small
    steps:
      # Keep your existing checkout and dependency-install steps.

      - name: Run E2E tests
        working-directory: tests/e2e
        run: npx shiplight test

      - name: Upload results to Shiplight
        if: always()
        working-directory: tests/e2e
        run: npx shiplight report

The highlighted lines are the only Shiplight-specific changes. Adjust working-directory if your tests live somewhere else.

Runner Sizes

Use runs-on to select the VM size for your job:

runs-on: shiplight-small

When the workflow is triggered, Shiplight provisions an ephemeral VM, registers it as a GitHub self-hosted runner, and destroys it after the job completes.

Label	vCPUs	Memory
shiplight-small	4	16 GB
shiplight-medium	8	32 GB
shiplight-large	16	64 GB
shiplight-xlarge	32	128 GB

No browser install needed

The runner image ships with Chromium and Playwright pre-installed. Do not run npx playwright install chromium in your workflow — it's already there.

What the Runner Provides

Shiplight runners come pre-configured with everything needed to run and report E2E tests:

• Action cache — when a cached locator self-heals during a CI run, the updated locator is persisted back to Shiplight Cloud so future runs replay at full speed without manual intervention.
• Chromium + Playwright — pre-installed, no npx playwright install step needed.
• Automatic result reporting — shiplight report works out of the box with no tokens or API URLs to configure. Credentials are provisioned per run and revoked automatically when the run completes.
• LLM access — if your tests use AI-powered actions (natural language steps, self-healing locators), the runner provides LLM credentials automatically. No API keys needed in your workflow.

Upload results

- name: Upload results to Shiplight
  if: always()
  working-directory: tests/e2e
  run: npx shiplight report

Uploads test results — including per-step screenshots, videos, and traces — to Shiplight Cloud. Results appear in the Test Results section, linked to your organization.

WARNING

Always use if: always() so results are uploaded even when tests fail. Without it, a red test run produces no cloud report.

On Shiplight runners, no additional configuration is needed — credentials are provided automatically.

On GitHub-hosted (non-Shiplight) runners

Shiplight also runs on stock ubuntu-latest runners, without the GitHub App. In your existing GitHub Actions workflow, make these changes:

Set SHIPLIGHT_API_TOKEN (an org token from app.shiplight.ai/api-tokens) at the job's env (global) scope. shiplight report needs it to authenticate the upload; shiplight test needs it only if your tests use AI-powered actions (natural-language steps, self-healing locators) that run through the Shiplight LLM proxy — deterministic YAML tests don't. Setting it at the job scope covers both cases. shiplight report additionally needs SHIPLIGHT_REPORT_TO_CLOUD=1 to enable the upload (Shiplight runners set this automatically; stock runners don't).

GitHub Actions sets CI=true, so the same token also enables the shared Shiplight Cloud action cache when it has action-cache:read and action-cache:write. Healed locators from passing tests are persisted for later runs; this cache is not limited to Shiplight-hosted runners.

jobs:
  e2e:
    runs-on: ubuntu-latest
    env: 
      SHIPLIGHT_API_TOKEN: ${{ secrets.SHIPLIGHT_API_TOKEN }}
    steps:
      # Keep your existing checkout, Node.js setup, dependency-install,
      # and Playwright browser-install steps.

      - name: Run E2E tests
        working-directory: tests/e2e
        run: npx playwright test
        run: npx shiplight test

      - name: Upload results to Shiplight
        if: always()
        working-directory: tests/e2e
        env: 
          SHIPLIGHT_REPORT_TO_CLOUD: "1"
        run: npx shiplight report

The highlighted lines are the only Shiplight-specific changes. Keep your existing stock-runner setup, including the Node.js and Playwright browser installation steps, and adjust working-directory if your tests live somewhere else.

Store the token as a repository or organization secret named SHIPLIGHT_API_TOKEN — never hardcode it in the workflow file, which is committed to your repo. For non-GitHub providers (GitLab, Jenkins, CircleCI, …), see the CI/CD overview.

Sharded runs (GHA matrix)

When using Playwright sharding with a GitHub Actions matrix, each shard produces its own report. Use shiplight report --merge to combine them into a single upload so the cloud dashboard shows one unified run instead of N separate ones.

Auto-triage CI failures (optional)

When an E2E workflow goes red, Shiplight can have an AI agent diagnose the failure and — for fixable spec issues — repair it automatically. The pipeline lives in the reusable workflow ShiplightAI/ci-triage. On a failed run it reads the run logs and the uploaded report artifacts (screenshots, traces), classifies each failure, and for failures it classifies as fixable spec issues it applies the fix, re-runs the test, and opens a PR. It never auto-merges.

The reusable workflow does not post to Slack, Linear, or any other system of yours. It publishes its diagnosis as an artifact; a job you own decides what to notify or file, using your own credentials (Step 3).

GitHub Actions only

Auto-triage is built on the GitHub workflow_run trigger and reusable workflows, so it is available on GitHub Actions only.

Set this up only once a test workflow (from the examples above) exists — triage triggers off that workflow's completion.

Step 1 — upload the report artifact from the test workflow

Triage reads failure evidence from a GitHub artifact, which is separate from the cloud upload. Add this step to your e2e.yml, after the test step:

- name: Upload test report (for triage)
  if: ${{ !cancelled() }}
  uses: ShiplightAI/ci-triage/upload-report@v1.1
  # sharded/matrix jobs: give each shard a unique name
  # with:
  #   name: test-report-shard-${{ matrix.shardIndex }}
  #   retention-days: "1"

The helper bakes in the shiplight-report/ path and drops the heavy Playwright traces (*.zip) and videos (*.webm) the agent never reads (~80% of the artifact size). Keep your npx shiplight report step too — the cloud report still gets full traces and videos for humans.

Step 2 — add the caller workflow

The workflow_run trigger and the per-repo credential mapping must live in your repo — a workflow_run trigger is illegal inside a reusable workflow, and secret names differ per repo. Everything else lives in the reusable workflow. Create .github/workflows/ci-failure-triage.yml:

name: CI Failure Triage

on:
  workflow_run:
    workflows: [E2E Tests] # exact `name:` of each test workflow to watch
    types: [completed]

jobs:
  triage:
    uses: ShiplightAI/ci-triage/.github/workflows/triage.yml@v1.1 # pin to a release tag, not @main
    permissions:
      contents: write
      pull-requests: write
      actions: read
    with:
      triage-runner: ubuntu-latest # read-only diagnosis job
      autofix-runner: shiplight-medium # re-runs tests, so needs browsers/network
      node-version: "22"
      allowed-paths: "tests templates" # top-level dirs the autofix agent may edit (hard guard)
    secrets:
      claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
      openai_api_key: ${{ secrets.OPENAI_API_KEY }} # Codex fallback when Claude is unavailable
      autofix_github_token: ${{ secrets.AUTOFIX_GITHUB_TOKEN }} # PAT/App token to open the PR; falls back to GITHUB_TOKEN
      # Per-repo credential mapping → generic env for the autofix re-run.
      # One KEY=VALUE per line; values must be single-line.
      extra_env: |
        BASE_URL=${{ vars.BASE_URL || 'https://example.com' }}
        MY_TEST_USER_PASSWORD=${{ secrets.MY_TEST_USER_PASSWORD }}
        MY_TEST_USER_2FA_SECRET=${{ secrets.MY_TEST_USER_2FA_SECRET }}

Notes:

• workflows: must list the exact name: of each test workflow to watch. Never list the triage workflow itself there.
• extra_env maps your repo's secret names onto the generic env the autofix job uses, so npx shiplight test and the MCP browser can authenticate during the re-run. Mirror the env: block from your test workflow.
• Provide at least one model credential (claude_code_oauth_token or anthropic_api_key); openai_api_key enables the Codex fallback. autofix_github_token is optional.
• autofix-runner re-runs the failing test, so it needs browsers and network — use a Shiplight runner, or install Chromium on a stock runner the same way your test workflow does.
• These examples assume the Shiplight project lives at the repository root. If it lives in a subdirectory, adjust the report path and add working-directory to the relevant steps.

WARNING

The triage job runs privileged — contents: write plus live credentials for the autofix re-run. Pin uses: to a release tag (@v1.1), never a moving branch like @main. For an immutable supply-chain pin, use the full commit SHA of the tag with the tag as a comment.

Step 3 — notify your own system (optional)

Triage publishes its diagnosis as the triage-context artifact: verdict.json (machine-readable) and triage.md (the human-readable write-up). Add a normal job you own to read it and call Slack, Linear, Jira, or anything else — your provider credentials stay in that job and are never visible to the triage agent.

publish-triage-incidents:
  needs: triage
  if: ${{ needs.triage.result != 'skipped' }}
  continue-on-error: true # an external system must never block triage or autofix
  runs-on: ubuntu-latest
  permissions:
    actions: read
    contents: read
  steps:
    - uses: actions/download-artifact@v4
      with:
        name: triage-context
        path: /tmp/triage-context
    - name: Publish to your system
      env:
        # Keep provider credentials here, never in `extra_env`.
        SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
        LINEAR_API_KEY: ${{ secrets.LINEAR_API_KEY }}
      run: jq '.failures[]' /tmp/triage-context/verdict.json

Each entry in verdict.json's failures[] has the test path, classification, confidence, fixable, a fix_summary, and a stable dedup_key. Pick your own policy — notify every failure, or file incidents only for fixable == false. For an issue tracker, key on dedup_key so a repeated failure updates the existing incident instead of opening a new one on every run.

Each autofix job additionally uploads an autofix-result-<test>-<index> artifact with the PR URL, verification status, and whether the fix was skipped. Read those in a later job if you want a separate post-autofix notification.

Troubleshooting

Job stays queued

• Confirm Shiplight Runners are enabled at app.shiplight.ai/runners, and check the Dispatches tab there for the failure reason. See CI Runners.
• Verify the Shiplight GitHub App is installed on the repository with the required permissions.
• Check whether your organization's runner concurrency limit has been reached. Contact your Shiplight admin to raise the limit if needed.

Test results not appearing in the dashboard

• Make sure the shiplight report step has if: always() so it runs after failures.
• Check the step logs for errors.

Chromium is downloading during CI

The runner has Chromium pre-installed. Do not add npx playwright install chromium to your workflow. If Playwright still downloads a browser, check that your package.json does not override the Playwright version bundled with shiplightai.

Related

• CI/CD overview — how the pieces fit together, and other providers
• CLI Reference — full shiplight test, shiplight report, and other command documentation
• Local Testing — project structure, authentication, and configuration