Self-Hosted Runners for CI

View as Markdown

Run Shiplight E2E tests on Shiplight-hosted GitHub Actions runners — ephemeral VMs with Chromium and Playwright pre-installed. Test results are automatically uploaded to Shiplight Cloud for trend tracking, flaky-test detection, and team visibility.

Prerequisites

1. Install the Shiplight GitHub App on your repository (or organization). The app requires Actions and Self-hosted runners permissions.

2. Enable Shiplight Runners — an org owner toggles the switch in Organization Settings.

3. Have a Shiplight test project in your repository. If you don't have one yet, scaffold it with:

   npx shiplightai@latest create tests/e2e
   cd tests/e2e && npm install

   See Local Testing for the full project setup guide.

Minimal Workflow

Create .github/workflows/e2e.yml:

name: E2E Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e:
    runs-on: shiplight-medium
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v5

      - name: Install dependencies
        working-directory: tests/e2e
        run: npm install

      - name: Run E2E tests
        working-directory: tests/e2e
        run: npx shiplight test

      - name: Upload results to Shiplight
        if: always()
        working-directory: tests/e2e
        run: npx shiplight report

That's it. The sections below explain each part in detail.

Runner Sizes

Use runs-on to select the VM size for your job:

runs-on: shiplight-medium

When the workflow is triggered, Shiplight provisions an ephemeral VM, registers it as a GitHub self-hosted runner, and destroys it after the job completes.

Label	vCPUs	Memory
shiplight-small	4	16 GB
shiplight-medium	8	32 GB
shiplight-large	16	64 GB
shiplight-xlarge	32	128 GB

No browser install needed

The runner image ships with Chromium and Playwright pre-installed. Do not run npx playwright install chromium in your workflow — it's already there.

What the Runner Provides

Shiplight runners come pre-configured with everything needed to run and report E2E tests:

• Chromium + Playwright — pre-installed, no npx playwright install step needed.
• Automatic result reporting — shiplight report works out of the box with no tokens or API URLs to configure. Credentials are provisioned per run and revoked automatically when the run completes.
• LLM access — if your tests use AI-powered actions (natural language steps, self-healing locators), the runner provides LLM credentials automatically. No API keys needed in your workflow.

Steps

Install dependencies

- name: Install dependencies
  working-directory: tests/e2e
  run: npm install

Installs shiplightai and its dependencies. Chromium is already on the runner — npm install is all you need.

Run tests

- name: Run E2E tests
  working-directory: tests/e2e
  run: npx shiplight test

shiplight test compiles .test.yaml files into Playwright specs and runs the full suite. All Playwright CLI flags are forwarded, so you can filter, shard, and customize as usual:

npx shiplight test --grep "checkout"            # run by name pattern
npx shiplight test --grep-invert "Stripe"       # exclude by pattern
npx shiplight test --project public             # run one Playwright project
npx shiplight test tests/login.test.yaml        # run one file

See CLI Reference for the full command documentation.

Upload results

- name: Upload results to Shiplight
  if: always()
  working-directory: tests/e2e
  run: npx shiplight report

Uploads test results — including per-step screenshots, videos, and traces — to Shiplight Cloud. Results appear in the Test Results section, linked to your organization.

WARNING

Always use if: always() so results are uploaded even when tests fail. Without it, a red test run produces no cloud report.

On Shiplight runners, no additional configuration is needed — credentials are provided automatically. For non-Shiplight runners or local uploads, see Report to Cloud.

Upload artifacts (optional)

- name: Upload test artifacts
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: e2e-report
    if-no-files-found: ignore
    path: |
      tests/e2e/shiplight-report
      tests/e2e/test-results

Uploads the HTML report, screenshots, videos, and traces as GitHub Actions Artifacts — useful for debugging directly from the GitHub UI.

Full Example

name: E2E Tests

on:
  push:
    branches: [main]
  pull_request:
  workflow_dispatch: {}

jobs:
  e2e:
    name: E2E regression tests
    runs-on: shiplight-medium
    timeout-minutes: 30
    env:
      BASE_URL: https://staging.example.com

    steps:
      - uses: actions/checkout@v5

      - name: Install dependencies
        working-directory: tests/e2e
        run: npm install

      - name: Run E2E tests
        id: tests
        continue-on-error: true
        working-directory: tests/e2e
        run: npx shiplight test

      - name: Upload results to Shiplight
        if: always() && steps.tests.outcome != 'skipped'
        working-directory: tests/e2e
        run: npx shiplight report

      - name: Upload artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: e2e-report
          if-no-files-found: ignore
          path: |
            tests/e2e/shiplight-report
            tests/e2e/test-results

      - name: Fail if tests failed
        if: steps.tests.outcome == 'failure'
        run: exit 1

The continue-on-error: true on the test step lets result upload and artifact collection proceed even when tests fail. The final step re-fails the job so the PR status check reflects the actual outcome.

Troubleshooting

Job stays queued

• Confirm Shiplight Runners are enabled in Org Settings.
• Verify the Shiplight GitHub App is installed on the repository with the required permissions.
• Check whether your organization's runner concurrency limit has been reached. Contact your Shiplight admin to raise the limit if needed.

Test results not appearing in the dashboard

• Make sure the shiplight report step has if: always() so it runs after failures.
• Check the step logs for errors.

Chromium is downloading during CI

The runner has Chromium pre-installed. Do not add npx playwright install chromium to your workflow. If Playwright still downloads a browser, check that your package.json does not override the Playwright version bundled with shiplightai.

Related

• CLI Reference — full shiplight test, shiplight report, and other command documentation
• Report to Cloud — uploading results from any CI provider (not just Shiplight runners)
• GitHub Actions Integration — trigger Shiplight Cloud test suites from GitHub Actions (a different integration path)
• Local Testing — project structure, authentication, and configuration