Test Results

View as Markdown

Once your tests upload results to Shiplight Cloud, every run is browsable at app.shiplight.ai with its steps, screenshots, video, and Playwright trace preserved. This page covers browsing and debugging individual results. For trends across many runs, see Analytics.

Getting results into the cloud

Results are uploaded by the shiplight report command after a test run:

npx shiplight test      # run the tests
npx shiplight report    # upload — run this even when tests fail

Two environment variables control the upload:

Variable	Purpose
SHIPLIGHT_API_TOKEN	Authenticates the upload and determines which organization owns the results
SHIPLIGHT_REPORT_TO_CLOUD	Must be exactly 1 to enable uploading

TIP

SHIPLIGHT_REPORT_TO_CLOUD must be the literal string 1 — true is not accepted, and if the variable is missing the upload is skipped silently with no error. On Shiplight CI runners both variables are injected for you.

The organization is always taken from the token, so a token decides where results land. See API Tokens.

Runs

app.shiplight.ai/runs lists every run, newest first, 20 per page.

Each run gets a per-organization sequential number displayed as RUN-42. Numbering starts at 1 for your org's first run and never reuses a number, so deleted runs leave permanent gaps.

Filter by repository, branch, status, trigger, author, CI provider, workflow, PR number, and date range. Filter options are built from your actual data, and the branch list narrows to whichever repository you picked.

The table's columns are configurable — show, hide, reorder, and resize them, and your layout is remembered in your browser. Available columns are result, tests, distribution, commit, branch, trigger, when, author, CI provider, workflow, and PR.

Run detail

Opening a run gives you a split view. The left panel is a tree of tests grouped by spec file, with a search box and a status filter. The right panel shows the selected test. Shiplight jumps straight to the first failing test when you open a run.

The right panel has four tabs:

• Steps — the full statement tree from the run, including nested conditionals, loops, and modal dismissals. Each step carries its status, duration, and any screenshot captured for it (click to zoom). Where a locator was auto-healed, you can see the healed action.
• Video — the recording of the test.
• Trace — the Playwright Trace Viewer, embedded inline. You can also download the trace or open it in a new tab.
• Info — details about the test itself.

Video and Trace are disabled when that artifact wasn't captured.

When a test was retried, an Attempt 1 / 2 / 3 selector appears so you can compare attempts.

The run info drawer, opened from the top right, carries the repository, branch, commit (with links to GitHub), commit message, PR, author, trigger, duration, and result counts, along with the run's LLM usage and estimated cost.

Tests

app.shiplight.ai/tests flips the view around: instead of one run at a time, it lists each unique test across all runs, sorted worst-first by pass rate.

Each row shows the latest result, pass rate, flake rate, run count, a sparkline of recent outcomes, and when it last ran. Filter by repository, date range, and status, or search by file and test name.

The pass rate and flake rate here are computed over each test's last 30 runs, not its whole history, so they reflect current behavior rather than lifetime averages.

Opening a test gives you two tabs:

• History — every run of that test, with result, duration, error, and available artifacts. Select any row to inspect it in place.
• Metrics — pass rate, flake rate, duration percentiles, a trend line, the most frequent error messages with counts, and a per-branch breakdown.

What counts as flaky

A test is flaky when it failed and then passed on retry within the same run. It is a recorded outcome from the test runner, not something Shiplight infers after the fact.

This matters when reading numbers:

• At the run level, a flaky test does not fail the run — it is counted with the passes, and the flaky count is tracked separately so the distribution bar can show it.
• At the test level, flaky counts against the pass rate: passed / (passed + flaky + failed).

Related

• Analytics — trends, flakiness, and failure attribution
• CI Runners — running tests on Shiplight-hosted runners
• API Tokens — tokens for uploading and reading results
• CI/CD — wiring the upload into your pipeline