Analytics
Analytics turns your uploaded runs into trends: whether your suite is getting healthier, which tests waste the most time, and whether failures are real regressions or noise. For individual runs, see Test Results.
Every view is scoped to your organization, and the run- and test-based views count only finished runs. The trend views can be filtered by repository, branch, and date range, with day, week, or month buckets.
Home dashboard
app.shiplight.ai/home is the summary: pass rate and volume cards, an executions chart, a pass-rate chart, and a Test Health card with three tabs — Failing, Flaky, and Slowest tests, top 10 each. Home picks its own bucket size from the date range rather than offering a selector.
Trend views
| View | Shows |
|---|---|
| Run Status | Passed and failed runs per bucket, with a pass-rate line |
| Run Duration | How long runs take — pick average, median, p95, min, or max |
| Test Status | Passed, flaky, failed, and skipped test executions over time |
| Test Flakiness | Flaky test count and flakiness rate; click a bar to drill into the tests |
| Failure Attribution | What actually caused your failures |
| Usage & Cost | What a test costs to run, and what drives that cost |
How the metrics are defined
Getting these right matters, because two reasonable-sounding "pass rates" are different numbers.
Run pass rate counts whole runs: passed / (passed + failed). Runs that errored or were cancelled are left out of both sides.
Test pass rate counts individual test executions: passed / (passed + flaky + failed). Timed-out tests count as failed, skipped tests are excluded entirely, and flaky counts against you.
Flakiness rate is flaky / (passed + flaky + failed) per bucket.
A test is flaky when it failed and then passed on retry inside the same run. It's an outcome reported by the test runner, not something inferred from history.
Why a run can pass with flaky tests in it
At the run level a flaky test is counted as a pass — it didn't break the build. At the test level it counts against the pass rate, because it did waste your time. Same data, two questions.
Run duration is the wall-clock time from run start to run end. Runs missing either timestamp are excluded.
The ranked tables
Failing and Flaky tests both require a test to have run at least 5 times in the window before it's ranked, so a single bad run doesn't dominate. Slowest tests are ranked on p50 and p95 duration.
A per-test flake rate that differs
On the Tests list, the flake rate column is a different measurement: it's how often a test changed outcome between consecutive runs, over its last 30 runs. It answers "how unstable is this test", where the analytics flakiness rate answers "how much of my execution volume was flaky". Expect the two numbers to differ for the same test.
Failure attribution
Attribution answers the question trends can't: was that failure our product breaking, or our test being wrong? It's built from classifications your triage workflow submits — typically the CI auto-triage pipeline.
| Category | Meaning |
|---|---|
| App Regression | The application actually broke — a real product regression |
| Spec / Test Issue | The test or spec is stale or wrong; the app behaved correctly |
| Test Data | A test-data or shared-account problem, not the app and not the test logic |
| Infra / Flake | Auth, infrastructure, or external flakiness |
| Unknown | Triage could not determine a cause |
The page shows each category's share of classified failures, plus an App Regression Rate trend — the single number that tells you whether your failures are getting more or less real over time.
This page only covers classified failures
It shows failures your triage workflow classified, not all failures. There is deliberately no coverage percentage, so treat the shares as composition among triaged failures rather than a complete picture.
Usage & cost
Usage & Cost answers what your suite costs to run, and where that cost comes from. Unlike Billing, which reports spend for the organization, this view is scaled per test — the number to watch when a suite grows.
It reports LLM cost per test, compute cost per test, and the two combined, alongside the levers that move them: tests run, average run time, LLM calls and tokens per test, and your cache hit rate. A low cache hit rate is usually the cheapest thing to fix, since cached locators replay without calling a model at all.
An LLM calls by operation table breaks the same usage down by what the agent was doing, with calls, total tokens, and the input/output split for each operation — so you can see which part of a test is expensive rather than just that it is.
Where the numbers suggest an obvious improvement, the page surfaces it as an optimization opportunity.
The date range is yours to pick; repository and bucket filters aren't available on this view yet.
Scheduled digests
Digests post a recurring summary to Slack. Configure them at app.shiplight.ai/notifications.
Choose a cadence — daily, weekly (Mondays), or monthly (the 1st), all sent at 08:00 UTC — and cover the period that just closed. Optionally scope a digest to a repository or branch.
Pick any combination of sections:
- Health — the summary for the period, with the change versus the previous one
- Flakiest — top flaky tests by flake count
- Most-failing — top failing tests
- Slowest — top tests by p95 duration
Ranked sections default to 10 entries and go up to 25. Unlike the analytics pages, digests include tests with as little as one execution, so short daily windows aren't empty. A period with nothing to report still sends an all-clear.
Reading analytics from the API
Everything here is available at https://nova-api.shiplight.ai/v1/analytics/* — summary, the four trend series, the ranked test lists, and attribution. Rates come back as percentages.
These endpoints require a token with the analytics:read scope, unlike the raw result endpoints which need no particular scope. See API Tokens.
Related
- Test Results — individual runs and per-test history
- API Tokens — scopes for the analytics API
- CI/CD — getting results and classifications flowing