Decide what to test
Use this workflow when you need to decide what to verify and which type of test to use. The goal is to provide sufficient evidence at a reasonable cost, rather than to maximize the number of tests.
Describe what changed
Provide a feature, specification, module, pull request, or ticket. For example:
/shiplight cover the checkout flow I just changedcover identifies the required coverage, selects a test strategy, creates and runs the tests, and records the session.
How Shiplight chooses a test strategy
Test types vary in cost and scope. cover selects the least expensive test type that can provide the required evidence:
| Modality | Cost | Strategic role |
|---|---|---|
| Unit | Very low | Use broadly for logic and edge cases |
| Contract / API | Low–med | Cover important public interfaces |
| Integration | Med | Verify database state and invariants that narrower tests cannot prove |
| E2E (codified YAML) | Med–high | Use for critical user journeys |
| Agent verification | High | Reserve for release gates that require judgment across the UI, API, DB, or logs |
Two principles guide the selection:
- Capability before cost. Cost is compared only among test types that can verify the behavior. For example, a unit test cannot verify a workflow in a real browser.
- Priority sets the minimum coverage. Each behavior uses the P0–P3 priority defined in your product requirements or specification. The available budget can add coverage, but does not reduce the minimum: every P0 behavior receives at least one release gate, and the strategy combines unit, integration, and E2E or agent verification where appropriate.
Review the test plan
The workflow creates two files for each feature:
specs/<feature>/test-spec.md— the behaviors and invariants to verify, each with its priorityspecs/<feature>/test-report.md— what was tested, by what test type, and what passed, failed, or was blocked
If the available budget does not support the preferred level of coverage, the report documents the selected approach and the remaining gap. This record allows reviewers to evaluate the decision without repeating the planning process.
A project can define its own strategy in a version-controlled TESTING.md file. If the file is not present, cover uses the built-in strategy.
Create the tests
cover runs the appropriate test-creation workflow:
/shiplight create-yaml-tests— Creates deterministic YAML E2E tests for critical user journeys./shiplight create-agent-verification— Creates agent verifications for release gates that must also verify backend state.
Run either command directly if you already know which test type you need. Use cover when the appropriate test strategy is not yet clear. See Create Tests for both workflows.