Shiplight CI Runners

View as Markdown

Run Shiplight E2E tests on Shiplight-hosted GitHub Actions runners — ephemeral VMs with Chromium and Playwright pre-installed. Test results are automatically uploaded to Shiplight Cloud for trend tracking, flaky-test detection, and team visibility.

Prerequisites

1. Install the Shiplight GitHub App on your repository (or organization). The app requires Actions and Self-hosted runners permissions.
2. Enable Shiplight Runners — an org owner toggles the switch in Organization Settings.

Minimal Workflow Example

Create .github/workflows/e2e.yml:

name: E2E Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e:
    runs-on: shiplight-small
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v5

      - name: Install dependencies
        working-directory: tests/e2e
        run: npm install

      - name: Run E2E tests
        working-directory: tests/e2e
        run: npx shiplight test

      - name: Upload results to Shiplight
        if: always()
        working-directory: tests/e2e
        run: npx shiplight report

That's it. The sections below explain each part in detail.

Runner Sizes

Use runs-on to select the VM size for your job:

runs-on: shiplight-small

When the workflow is triggered, Shiplight provisions an ephemeral VM, registers it as a GitHub self-hosted runner, and destroys it after the job completes.

Label	vCPUs	Memory
shiplight-small	4	16 GB
shiplight-medium	8	32 GB
shiplight-large	16	64 GB
shiplight-xlarge	32	128 GB

No browser install needed

The runner image ships with Chromium and Playwright pre-installed. Do not run npx playwright install chromium in your workflow — it's already there.

What the Runner Provides

Shiplight runners come pre-configured with everything needed to run and report E2E tests:

• Action cache — when a cached locator self-heals during a CI run, the updated locator is persisted back to Shiplight Cloud so future runs replay at full speed without manual intervention.
• Chromium + Playwright — pre-installed, no npx playwright install step needed.
• Automatic result reporting — shiplight report works out of the box with no tokens or API URLs to configure. Credentials are provisioned per run and revoked automatically when the run completes.
• LLM access — if your tests use AI-powered actions (natural language steps, self-healing locators), the runner provides LLM credentials automatically. No API keys needed in your workflow.

Upload results

- name: Upload results to Shiplight
  if: always()
  working-directory: tests/e2e
  run: npx shiplight report

Uploads test results — including per-step screenshots, videos, and traces — to Shiplight Cloud. Results appear in the Test Results section, linked to your organization.

WARNING

Always use if: always() so results are uploaded even when tests fail. Without it, a red test run produces no cloud report.

On Shiplight runners, no additional configuration is needed — credentials are provided automatically.

On non-Shiplight runners

shiplight report also works on any other CI provider (GitHub-hosted runners, GitLab, CircleCI, etc.). You need to set two environment variables: SHIPLIGHT_REPORT_TO_CLOUD=1 to enable the upload (Shiplight runners set this automatically; other runners don't), and SHIPLIGHT_API_TOKEN to an org API token from nova.shiplight.ai/api-tokens:

- name: Upload results to Shiplight
  if: always()
  working-directory: tests/e2e
  env:
    SHIPLIGHT_REPORT_TO_CLOUD: "1"
    SHIPLIGHT_API_TOKEN: ${{ secrets.SHIPLIGHT_API_TOKEN }}
  run: npx shiplight report

Store the token as a repository or organization secret in your CI provider, then reference it as shown above.

Sharded runs (GHA matrix)

When using Playwright sharding with a GitHub Actions matrix, each shard produces its own report. Use shiplight report --merge to combine them into a single upload so the cloud dashboard shows one unified run instead of N separate ones.

Troubleshooting

Job stays queued

• Confirm Shiplight Runners are enabled in Org Settings.
• Verify the Shiplight GitHub App is installed on the repository with the required permissions.
• Check whether your organization's runner concurrency limit has been reached. Contact your Shiplight admin to raise the limit if needed.

Test results not appearing in the dashboard

• Make sure the shiplight report step has if: always() so it runs after failures.
• Check the step logs for errors.

Chromium is downloading during CI

The runner has Chromium pre-installed. Do not add npx playwright install chromium to your workflow. If Playwright still downloads a browser, check that your package.json does not override the Playwright version bundled with shiplightai.

Related

• CLI Reference — full shiplight test, shiplight report, and other command documentation
• GitHub Actions Integration — trigger Shiplight Cloud test suites from GitHub Actions (a different integration path)
• Local Testing — project structure, authentication, and configuration