Report to Cloud

View as Markdown

After a local test run, the Shiplight CLI can upload results to Shiplight Cloud — including per-step screenshots, video recordings, traces, and CI / git metadata. This lets you view and share results from local or CI runs alongside cloud-executed tests, without moving test execution to the cloud.

Cloud upload is a separate step after shiplight test — handled by shiplight report. That one command covers both single-process runs and merged sharded runs, so every setup uses the same code path.

Setup

1. Get an API token

Get your API token from app.shiplight.ai/settings/api-tokens. Store it in your secret manager (GitHub Actions secrets, CI vault, etc.) — never commit it to the repo.

2. Run shiplight report after shiplight test

After shiplight test writes its local shiplight-report/ directory, run shiplight report with REPORT_TO_CLOUD=true and SHIPLIGHT_API_TOKEN in the environment. The same command works for a single-process run and for merging multiple shards.

Single-process run:

npx shiplight test
REPORT_TO_CLOUD=true SHIPLIGHT_API_TOKEN=sk-... npx shiplight report

Sharded run (Playwright shards, merged after all complete):

# Shard 1..N (in parallel)
npx shiplight test --shard=$SHARD/$TOTAL

# Merge step (after all shards finish)
REPORT_TO_CLOUD=true SHIPLIGHT_API_TOKEN=sk-... \
  npx shiplight report --merge all-shards/*/shiplight-report/

Both modes upload a single combined run to the cloud. After the upload completes, the CLI logs the cloud report URL:

Shiplight cloud report: https://app.shiplight.ai/runs/12345

The SHIPLIGHT_API_TOKEN env var is only read by shiplight report. shiplight test itself has no cloud-upload path — running it without the token is fine and produces the same local HTML report as before.

What gets uploaded

For each test:

Asset	Description
Per-step screenshots	One screenshot per YAML step, matched to step descriptions
Video	Full test recording (if enabled in Playwright config)
Trace	Playwright trace archive (if enabled)
Report JSON	Step-by-step results including status, duration, and AI messages

Alongside the results, shiplight report collects CI and git metadata automatically — see CI metadata below.

Environment variables

Variable	Required?	Description
REPORT_TO_CLOUD	Yes ("true")	Gate the upload. shiplight report is a no-op for cloud unless this is set
SHIPLIGHT_API_TOKEN	Yes (when REPORT_TO_CLOUD=true)	Bearer token from app.shiplight.ai/settings/api-tokens
SHIPLIGHT_API_URL	Optional	Override the Shiplight API base URL (default https://api.shiplight.ai)

CI metadata

shiplight report automatically collects git and CI metadata and attaches it to the cloud run. No configuration is needed — it reads from standard CI environment variables.

Collected fields

Field	Description
Commit SHA	The HEAD commit of the branch being tested
Branch	Source branch name
PR number	Pull request number (if triggered from a PR)
PR title	Pull request title
Commit message	Subject line of the HEAD commit
Author email	Commit author
CI build ID & URL	Link back to the CI run
Commit URL	Direct link to the commit on GitHub / GitLab

Environment variable overrides

Some CI setups — particularly repository_dispatch triggers where the workflow runs on the default branch rather than the PR branch — require explicit overrides. Set these environment variables to supply values shiplight report cannot derive automatically:

Variable	Description
SHIPLIGHT_GIT_SHA	Override the commit SHA (e.g. the actual PR commit, not the merge commit)
SHIPLIGHT_PR_NUMBER	Override the PR number
SHIPLIGHT_PR_TITLE	Override the PR title
SHIPLIGHT_GIT_BRANCH	Override the branch name

These take precedence over any automatically detected values.

GitHub Actions

Single-process run

- name: Run tests
  run: npx shiplight test

- name: Upload report to Shiplight Cloud
  if: always()
  env:
    REPORT_TO_CLOUD: "true"
    SHIPLIGHT_API_TOKEN: ${{ secrets.SHIPLIGHT_API_TOKEN }}
  run: npx shiplight report

Running the upload in a separate step with if: always() lets the step fire whether or not the test run passed — you'll get a cloud report for red runs too.

Sharded run

jobs:
  test:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - name: Run shard
        run: npx shiplight test --shard=${{ matrix.shard }}/4
      - name: Upload shard report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: shiplight-report-shard-${{ matrix.shard }}
          path: shiplight-report/

  merge-and-upload:
    needs: test
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          pattern: shiplight-report-shard-*
          path: all-shards
      - name: Merge and upload to Shiplight Cloud
        env:
          REPORT_TO_CLOUD: "true"
          SHIPLIGHT_API_TOKEN: ${{ secrets.SHIPLIGHT_API_TOKEN }}
        run: npx shiplight report --merge all-shards/*/shiplight-report/

The merge step downloads all shard artifacts, combines them into a single report, and uploads one unified run to Shiplight Cloud. Only the merge job needs SHIPLIGHT_API_TOKEN — the parallel shard jobs are pure test execution with no cloud credentials.

With Vercel Preview deployments

When using repository_dispatch or deployment_status events triggered by Vercel, GITHUB_SHA points to the default branch — not the PR commit. Resolve the correct metadata ahead of time and pass it via the override env vars:

- name: Resolve PR metadata
  id: resolve
  uses: actions/github-script@v7
  with:
    script: |
      const sha = context.payload.client_payload?.git?.sha
                  ?? context.payload.deployment?.sha;
      const branch = context.payload.deployment?.ref;
      const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
        owner: context.repo.owner,
        repo: context.repo.repo,
        commit_sha: sha,
      });
      core.setOutput('git_sha', sha ?? '');
      core.setOutput('git_branch', branch ?? prs[0]?.head?.ref ?? '');
      core.setOutput('pr_number', String(prs[0]?.number ?? ''));
      core.setOutput('pr_title', prs[0]?.title ?? '');

- name: Run tests
  env:
    PLAYWRIGHT_BASE_URL: ${{ github.event.client_payload.url || github.event.deployment_status.environment_url }}
  run: npx shiplight test

- name: Upload report to Shiplight Cloud
  if: always()
  env:
    REPORT_TO_CLOUD: "true"
    SHIPLIGHT_API_TOKEN: ${{ secrets.SHIPLIGHT_API_TOKEN }}
    SHIPLIGHT_GIT_SHA: ${{ steps.resolve.outputs.git_sha }}
    SHIPLIGHT_GIT_BRANCH: ${{ steps.resolve.outputs.git_branch }}
    SHIPLIGHT_PR_NUMBER: ${{ steps.resolve.outputs.pr_number }}
    SHIPLIGHT_PR_TITLE: ${{ steps.resolve.outputs.pr_title }}
  run: npx shiplight report