YAML E2E Tests

View as Markdown

Why YAML E2E Tests?

🔄 Self-healing — no brittle selectors. Steps are cached for speed, but when a locator breaks the AI re-reads the page and resolves the intent automatically. No flaky tests, no selector maintenance.

💬 Readable in natural language. Every step describes what should happen, not how. Agents can author and maintain the tests, while humans can review them like specs and adjust details when needed.

YAML E2E tests are end-to-end browser tests authored in YAML instead of Playwright code. They are designed for coding agents to create and maintain, with humans in control through readable diffs, local runs, and the visual debugger. Shiplight tests are intent-driven: every step is a natural language description of what should happen. The AI reads the page and figures out the rest. For speed, steps can be enriched with action: or js: caches that replay deterministically (<1s), with automatic AI fallback when locators go stale (self-healing).

Agents author these — /shiplight create-yaml-tests

You rarely write this YAML by hand. Run /shiplight create-yaml-tests and your agent authors and maintains the tests by walking your app, and /shiplight fix to repair them when they drift. This page documents the format so you can review the results like a spec and hand-tune complex flows in the visual debugger.

Full Spec & Examples

For the complete language specification, see the YAML E2E Test Language Spec. For ready-to-run examples, see the examples repo.

No lock-in: YAML E2E tests can be run directly with the Shiplight CLI (shiplightai), or transpiled to standard Playwright test files that run independently, fully compatible, no runtime dependency. You can eject at any time.

Basic Test

Every line is a natural language instruction. The AI resolves each one at runtime by looking at the page and performing the right action.

goal: Verify user can create a new project
base_url: https://app.example.com

statements:
  - URL: /projects
  - intent: Click the "New Project" button
  - intent: Enter "My Test Project" in the project name field
  - intent: Select "Public" from the visibility dropdown
  - intent: Click "Create"
  - VERIFY: Project page shows title 'My Test Project'
teardown:
  - intent: Delete the created project

Enriched Test

After exploring the UI with Shiplight MCP tools, the coding agent enriches natural language steps with action caches for deterministic, fast replay:

goal: Verify user can create a new project
base_url: https://app.example.com

statements:
  - URL: /projects
  - STEP: Create project
    statements:
      - intent: Click the New Project button
        action: click
        locator: "getByRole('button', { name: 'New Project' })"
      - intent: Enter "My Test Project" in the project name field
        action: input_text
        text: "My Test Project"
        locator: "getByRole('textbox', { name: 'Project name' })"
      - intent: Click Create
        action: click
        locator: "getByRole('button', { name: 'Create' })"
  - VERIFY: Project page shows title 'My Test Project'
teardown:
  - intent: Delete the created project

• ACTION statements (action: or js:, <1s each) — fast deterministic replay, with automatic AI fallback if the locator fails (self-healing)
• VERIFY statements — AI-powered natural language assertions. Can include js: to speed up simple checks, with automatic fallback to AI verification
• DRAFT statements (natural language, ~5-10s each) — the AI reads the page and figures out what to do. Used for steps not yet enriched with action caches

See Test Structure for the full breakdown of the test file, and Statement Types for every kind of step you can write.

Locators Are a Cache

Locators are a performance cache, not a hard dependency. When the UI changes and a locator becomes stale, Shiplight's agentic layer auto-heals by falling back to the natural language description to find the right element.

However, when a locator is permanently changed (e.g., a button was renamed or moved), the cached locator will fail on every run: self-heal recovers that run, but the YAML in your repo doesn't update automatically. Run /shiplight fix to refresh the cached locator so future runs replay at full speed again. For the full mechanism, including what gets persisted where and which steps can heal, see How self-healing works.

The Enrichment Workflow

1. Draft — The agent writes tests in natural language (DRAFT statements)
2. Explore — The agent uses inspect_page and act to walk through the UI
3. Collect — The agent uses get_locators to capture element locators and Playwright code
4. Enrich — The agent replaces DRAFT steps with ACTION statements (intent: + action:/locator:)
5. Result — Tests run 10x faster with deterministic replay

DRAFT and ACTION statements can be mixed in the same test. The agent starts with all natural language, then selectively enriches the most-used flows.