Agent Verification

View as Markdown

An agent verification exercises a complete user journey and then proves the system actually did what the screen claimed. It drives the front end, then checks the backend it should have changed — API responses, database rows, audit events, logs, queues.

It is the most comprehensive format Shiplight offers, and the right one for smoke-testing core user journeys before a release.

Why it catches more

Every other format sees one layer:

Format	Sees	Misses
E2E / UI test	What the screen shows	Whether the system actually recorded it
API test	One service's responses	Whether the user can reach it, and what else it touched
Contract test	The shape of a payload	Whether the real journey produces that payload
Agent verification	The journey and the state it left behind	—

A checkout that renders "Order confirmed" but never writes the order row passes a UI test and fails an agent verification. So does a signup that creates the account but skips the audit event, or an upgrade that charges the card and leaves the plan unchanged. These are the failures that reach production precisely because each individual test layer was green.

What it costs

This power is not free, and the cost is why you run a handful of these rather than hundreds:

• Minutes per case, not seconds. A case walks a full journey and queries several systems.
• Model cost on every run. A coding agent executes the case, so each run consumes tokens.
• Judgment, not bit-for-bit determinism. The verdict comes from an agent weighing evidence it collected. It is auditable — every claim is backed by a report, query result, or trace — but two runs are not byte-identical.
• Real access required. The case needs credentials and reachable database, log, and API endpoints for its target environment.

When to use it

Smoke-test the journeys that must not break: signup, login, checkout, payment, provisioning, anything touching money or data integrity. Run them before a release, or against staging and production after a deploy.

Keep the set small and high-value. For everything else, a YAML E2E test is cheaper and faster, and belongs on every pull request.

Authoring a case

/shiplight create an agent verification for the checkout journey —
the order should land in the database, not just show a confirmation

Natural phrasing works; you don't need the exact subcommand token. Don't open with the word "verify", though — that routes to the one-off verify check instead.

A case is a Markdown file describing the requirement under test, the environments it can run against, the fixtures it needs, the checks to perform across each layer, the evidence each check must produce, and what to clean up afterward.

Two rules matter more than the rest. Environment preflight is separate from verification — if the environment can't execute the case, it must stop rather than report a product failure. And every assertion is unconditional for state the case creates itself; a check written as "if the UI offers X, verify Y" quietly passes whenever X isn't found, dropping the coverage the case exists to provide.

Running

Cases live under tests/agent/ and are grouped into suites by a manifest:

{
  "smoke": [{ "id": "checkout", "case": "tests/agent/checkout/purchase.md", "required": true, "timeout_minutes": 30 }]
}

Run the agent verification runner directly for a suite or a single case:

npx tsx tests/agent/run-agent-verification.ts --target staging --suite smoke
npx tsx tests/agent/run-agent-verification.ts --target local --case tests/agent/checkout/purchase.md

Cases are portable across environments — the same case runs against local, staging, and production, reading each target's URLs, credentials, and evidence sources from the harness.

Reading the result

Each run produces a report with the evidence it collected, ending in one status:

• PASS / FAIL — the case ran and reached a verdict. A failed check is a finding.
• BLOCKED — the environment couldn't execute the case: no database access, unreachable URL, failed login, missing fixtures, or a route or table that no longer exists. Not a product failure, and never reported as one.
• ABORTED — the run was interrupted. Release gates ignore it and rerun.

The FAIL versus BLOCKED split is the one to understand: a check that ran and failed tells you something about your product, while a check that couldn't run tells you nothing at all. Required cases fail the suite unless they return PASS.

Adopting it

The runner and case template are scaffolded into your repo the first time you author a case, and become yours to own — along with the manifest, fixtures, credentials, and CI wiring.