Self-hosting

View as Markdown

Run the Shiplight CLI and browsers on your own machine or CI runners, bring your own AI provider key (BYOK), and keep action cache and test reports in storage you control. This setup requires no Shiplight account or API token.

1. Bring your own key

Follow Bring your own key to choose a provider. Put its key in your project's ignored .env file, or inject it through your CI secret store. For example:

GOOGLE_API_KEY=your-google-api-key
WEB_AGENT_FALLBACK_MODELS=
SHIPLIGHT_TELEMETRY=0

Remove SHIPLIGHT_API_TOKEN from both .env and the runner environment for this setup. The provider bills AI usage to your account. The empty WEB_AGENT_FALLBACK_MODELS disables the built-in cross-provider fallback chain; SHIPLIGHT_TELEMETRY=0 disables the CLI's anonymous usage telemetry.

See Choosing a model for model and fallback settings, and OpenAI-compatible endpoints to connect your own model service. BYOK sends the page context needed for AI actions to your selected provider.

2. Persist local action cache in CI

Shiplight saves self-healed actions from passing tests so later runs can replay the repaired locators. Set SHIPLIGHT_ACTION_CACHE_BACKEND=local to use the local action cache: Shiplight reads cached actions from .shiplight/action-cache/ and writes updates back to that directory. Restore and save the directory using your CI provider's cache storage to reuse it across runs.

Upgrade the Shiplight CLI to the latest version before using this setup. For an npm-based test project, run:

npm install --save-dev shiplightai@latest

Commit the updated package.json and lockfile so CI installs the same version with npm ci.

GitHub Actions example

This workflow assumes the test project is at the repository root. Add GOOGLE_API_KEY as a repository Actions secret, or substitute your chosen provider's key. Add any application credentials your tests need as separate secrets.

Save this as .github/workflows/e2e-cache.yml:

name: E2E with BYOK and local action cache

on:
  workflow_dispatch:

permissions:
  contents: read

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: false

jobs:
  e2e:
    runs-on: ubuntu-latest
    env:
      SHIPLIGHT_ACTION_CACHE_BACKEND: local
      GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
      WEB_AGENT_FALLBACK_MODELS: ""
      SHIPLIGHT_TELEMETRY: "0"
      SHIPLIGHT_REPORT_DIR: shiplight-report
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
      - run: npm ci
      - run: npx playwright install --with-deps chromium

      - name: Restore action cache
        id: action-cache
        uses: actions/cache/restore@v4
        with:
          path: .shiplight/action-cache
          key: shiplight-action-v1-${{ github.workflow }}-${{ github.ref }}-${{ github.run_id }}-${{ github.run_attempt }}
          restore-keys: |
            shiplight-action-v1-${{ github.workflow }}-${{ github.ref }}-

      - name: Run tests
        id: tests
        run: npx shiplight test

      - name: Save action cache
        if: ${{ !cancelled() && steps.tests.outcome != 'skipped' && hashFiles('.shiplight/action-cache/*.json') != '' }}
        uses: actions/cache/save@v4
        with:
          path: .shiplight/action-cache
          key: ${{ steps.action-cache.outputs.cache-primary-key }}

      - name: Upload test report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: shiplight-report
          path: shiplight-report/

      - name: Token usage summary
        if: ${{ !cancelled() && hashFiles('shiplight-report/report-data.json') != '' }}
        run: |
          jq -r '
            if ((.usageSummary.by_operation // []) | length) > 0 then
              "### LLM token usage",
              "| Operation | Model | Calls | Input | Output | Thinking | Cache read |",
              "|---|---|--:|--:|--:|--:|--:|",
              (.usageSummary.by_operation[] | "| \(.operation) | \(.model) | \(.calls) | \(.input_tokens) | \(.output_tokens) | \(.thinking_tokens) | \(.cache_read_tokens) |"),
              (.usageSummary.by_operation | "| **Total** | | \(map(.calls)|add) | \(map(.input_tokens)|add) | \(map(.output_tokens)|add) | \(map(.thinking_tokens)|add) | \(map(.cache_read_tokens)|add) |")
            else "No LLM usage reported for this run." end
          ' shiplight-report/report-data.json >> "$GITHUB_STEP_SUMMARY"

The save step also runs when some tests fail: passing tests may have produced useful repairs. Shiplight writes new cache entries only for tests whose final status is passed.

Understand the cache key

The key contains:

• shiplight-action-v1: a namespace and manually managed cache version. Bump v1 to start a fresh cache.
• github.workflow: the workflow name, isolating workflows with different names.
• github.ref: the branch or other Git ref.
• github.run_id: one triggered execution, not the workflow definition's ID.
• github.run_attempt: the attempt number; rerunning a run keeps its run ID and increments this value.

Renaming a workflow changes its cache prefix. Each run/attempt saves a new key; restoration uses the workflow-and-ref prefix to find a previous archive. For the immutable-key and restore-prefix behavior, see the GitHub cache action.

For matrix jobs, include a project/shard identifier in both the save key and restore prefix. Each shard then owns its cache. A shared archive does not automatically merge concurrent shard results.

Runner and report storage

The example uses GitHub-hosted ubuntu-latest. To execute on your own registered Linux runner, change runs-on to [self-hosted, linux, x64] and ensure it has the browser dependencies and network access your tests need. GitHub cache storage still lives on GitHub; use your internal cache service if all storage must remain on your infrastructure.

SHIPLIGHT_REPORT_DIR writes the report to a fixed shiplight-report/ folder, and the upload step keeps it as a CI artifact, even when tests fail. The folder contains the HTML report with each test's screenshots, video, and trace. To keep reports on your own infrastructure, copy this folder to your internal storage instead.

View a test report

Download and unzip the shiplight-report artifact. From your test project, run:

npx shiplight report path/to/shiplight-report --open

This rebuilds the HTML for your machine and opens it in your browser. Opening index.html directly also works, but the trace links then point to paths on the CI runner. To open a trace, run npx playwright show-trace path/to/shiplight-report/test-0-trace.zip.

shiplight report uploads to Shiplight Cloud only when both SHIPLIGHT_REPORT_TO_CLOUD and SHIPLIGHT_API_TOKEN are set. Without them, nothing leaves your machine.

Traces record the values tests type, including credentials. Restrict access to report artifacts as you would any secret.

View token usage

The token usage summary step adds a table of LLM calls and tokens per operation and model to the run's summary page. The data comes from usageSummary in shiplight-report/report-data.json. "No LLM usage reported" means the run made no LLM calls, for example when every step replayed from the action cache. The step needs jq, which GitHub-hosted runners include; install it on self-hosted runners.