--- title: "Analytics" description: "Trends, flakiness, and failure attribution in Shiplight Cloud — what each metric means, how it's calculated, and how to get scheduled digests in Slack." --- # Analytics
Analytics turns your uploaded runs into trends: whether your suite is getting healthier, which tests waste the most time, and whether failures are real regressions or noise. For individual runs, see [Test Results](/cloud_v2/test-results). Every view is scoped to your organization, and the run- and test-based views count only finished runs. The trend views can be filtered by repository, branch, and date range, with day, week, or month buckets. ## Home dashboard [app.shiplight.ai/home](https://app.shiplight.ai/home) is the summary: pass rate and volume cards, an executions chart, a pass-rate chart, and a **Test Health** card with three tabs — **Failing**, **Flaky**, and **Slowest** tests, top 10 each. Home picks its own bucket size from the date range rather than offering a selector. ## Trend views | View | Shows | | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------ | | [Run Status](https://app.shiplight.ai/analytics/run-status) | Passed and failed runs per bucket, with a pass-rate line | | [Run Duration](https://app.shiplight.ai/analytics/run-duration) | How long runs take — pick average, median, p95, min, or max | | [Test Status](https://app.shiplight.ai/analytics/test-status) | Passed, flaky, failed, and skipped test executions over time | | [Test Flakiness](https://app.shiplight.ai/analytics/test-flakiness) | Flaky test count and flakiness rate; click a bar to drill into the tests | | [Failure Attribution](https://app.shiplight.ai/analytics/failure-attribution) | What actually caused your failures | | [Usage & Cost](https://app.shiplight.ai/analytics/usage) | What a test costs to run, and what drives that cost | ## How the metrics are defined Getting these right matters, because two reasonable-sounding "pass rates" are different numbers. **Run pass rate** counts whole runs: `passed / (passed + failed)`. Runs that errored or were cancelled are left out of both sides. **Test pass rate** counts individual test executions: `passed / (passed + flaky + failed)`. Timed-out tests count as failed, skipped tests are excluded entirely, and **flaky counts against you**. **Flakiness rate** is `flaky / (passed + flaky + failed)` per bucket. A test is **flaky** when it failed and then passed on retry inside the same run. It's an outcome reported by the test runner, not something inferred from history. ::: tip Why a run can pass with flaky tests in it At the run level a flaky test is counted as a pass — it didn't break the build. At the test level it counts against the pass rate, because it did waste your time. Same data, two questions. ::: **Run duration** is the wall-clock time from run start to run end. Runs missing either timestamp are excluded. ### The ranked tables **Failing** and **Flaky** tests both require a test to have run at least **5 times** in the window before it's ranked, so a single bad run doesn't dominate. **Slowest** tests are ranked on p50 and p95 duration. ### A per-test flake rate that differs On the [Tests](https://app.shiplight.ai/tests) list, the flake rate column is a different measurement: it's how often a test _changed outcome_ between consecutive runs, over its last 30 runs. It answers "how unstable is this test", where the analytics flakiness rate answers "how much of my execution volume was flaky". Expect the two numbers to differ for the same test. ## Failure attribution Attribution answers the question trends can't: _was that failure our product breaking, or our test being wrong?_ It's built from classifications your triage workflow submits — typically the [CI auto-triage pipeline](/local/ci/github-actions#auto-triage-ci-failures-optional). | Category | Meaning | | --------------------- | ------------------------------------------------------------------------- | | **App Regression** | The application actually broke — a real product regression | | **Spec / Test Issue** | The test or spec is stale or wrong; the app behaved correctly | | **Test Data** | A test-data or shared-account problem, not the app and not the test logic | | **Infra / Flake** | Auth, infrastructure, or external flakiness | | **Unknown** | Triage could not determine a cause | The page shows each category's share of classified failures, plus an **App Regression Rate** trend — the single number that tells you whether your failures are getting more or less real over time. ::: warning This page only covers classified failures It shows failures your triage workflow classified, not all failures. There is deliberately no coverage percentage, so treat the shares as composition among triaged failures rather than a complete picture. ::: ## Usage & cost [Usage & Cost](https://app.shiplight.ai/analytics/usage) answers what your suite costs to run, and where that cost comes from. Unlike [Billing](/cloud_v2/billing), which reports spend for the organization, this view is scaled per test — the number to watch when a suite grows. It reports **LLM cost per test**, **compute cost per test**, and the two combined, alongside the levers that move them: tests run, average run time, LLM calls and tokens per test, and your **cache hit rate**. A low cache hit rate is usually the cheapest thing to fix, since cached locators replay without calling a model at all. An **LLM calls by operation** table breaks the same usage down by what the agent was doing, with calls, total tokens, and the input/output split for each operation — so you can see which part of a test is expensive rather than just that it is. Where the numbers suggest an obvious improvement, the page surfaces it as an **optimization opportunity**. The date range is yours to pick; repository and bucket filters aren't available on this view yet. ## Scheduled digests Digests post a recurring summary to Slack. Configure them at [app.shiplight.ai/notifications](https://app.shiplight.ai/notifications). Choose a cadence — **daily**, **weekly** (Mondays), or **monthly** (the 1st), all sent at 08:00 UTC — and cover the period that just closed. Optionally scope a digest to a repository or branch. Pick any combination of sections: - **Health** — the summary for the period, with the change versus the previous one - **Flakiest** — top flaky tests by flake count - **Most-failing** — top failing tests - **Slowest** — top tests by p95 duration Ranked sections default to 10 entries and go up to 25. Unlike the analytics pages, digests include tests with as little as one execution, so short daily windows aren't empty. A period with nothing to report still sends an all-clear. ## Reading analytics from the API Everything here is available at `https://nova-api.shiplight.ai/v1/analytics/*` — summary, the four trend series, the ranked test lists, and attribution. Rates come back as percentages. These endpoints require a token with the **`analytics:read`** scope, unlike the raw result endpoints which need no particular scope. See [API Tokens](/cloud_v2/api-tokens). ## Related - [Test Results](/cloud_v2/test-results) — individual runs and per-test history - [API Tokens](/cloud_v2/api-tokens) — scopes for the analytics API - [CI/CD](/local/ci) — getting results and classifications flowing