Local Testing with Shiplight CLI

View as Markdown

The Shiplight CLI (shiplightai) runs YAML E2E tests locally: end-to-end browser tests authored in YAML instead of Playwright code. Coding agents create and maintain these tests, while humans can review the readable YAML, run the tests locally, and hand-tune complex UI flows in the debugger. YAML E2E tests run alongside your existing .test.ts files with no separate tooling.

The CLI includes an AI agent for natural language actions, self-healing locators, and VERIFY assertions.

Let your agent set this up

You rarely need to scaffold or wire this by hand. Run /shiplight init to scaffold the project and /shiplight create-yaml-tests to author tests: your coding agent handles dependencies, API keys, playwright.config.ts, and the first tests. This page documents what those produce so you can review, run, and hand-tune the results. See the full /shiplight skill.

Prerequisites

• Node.js >= 22
• AI API key — GOOGLE_API_KEY (Get key) or ANTHROPIC_API_KEY (Get key). A Shiplight API token works too, via the LLM proxy — see LLM Providers for all options.

Store API keys and credentials in a .env file in your project root: the CLI auto-discovers it on startup. Make sure .env is in your .gitignore.

Quick Start

Invoke /shiplight create-yaml-tests in your coding agent (Claude Code, Cursor, or Codex) to get started. The agent scaffolds the project, installs dependencies, configures API keys, and writes the first YAML E2E tests for your app.

Run tests with:

npx shiplight test

Project Structure

A typical project follows standard Playwright conventions. Shiplight adds .env for API keys and credentials.

my-tests/
├── playwright.config.ts
├── package.json
├── .env                            # API keys + credentials (gitignored)
├── .gitignore
│
├── tests/
│   ├── public-app/                 # No login needed
│   │   ├── search.test.yaml
│   │   └── filter.test.yaml
│   │
│   └── my-saas-app/               # Requires login
│       ├── auth.setup.ts           # Playwright login setup
│       ├── dashboard.test.yaml
│       └── settings.test.yaml

Run all tests:

npx shiplight test

Run one project:

npx shiplight test my-saas-app/

After the run completes, open shiplight-report/index.html to view the results. See Report for customization.

YAML E2E Test Format

See YAML E2E Test Format for a quick overview of statement types, actions, conditionals, loops, variables, and templates.

For the complete language specification and ready-to-run examples, see the examples repo.

Authentication (Optional)

If your app requires login, run /shiplight auth and describe your login flow: your agent sets up (or repairs) the login and saved storage state. See Authentication for the shared-account and per-test patterns it uses.

Configuration

The shiplightConfig() helper in playwright.config.ts handles YAML transpilation and sets up the Shiplight reporter. It returns a partial Playwright config that you spread into defineConfig:

import { defineConfig, shiplightConfig } from "shiplightai";

export default defineConfig({
  ...shiplightConfig(),
  // Your Playwright config (testDir, projects, use, etc.)
});

Report

shiplightConfig() automatically enables the Shiplight HTML reporter. No extra configuration needed. After each run, open shiplight-report/index.html to view results with per-step screenshots, videos, and traces.

To customize the report output, override the reporter field in your config:

export default defineConfig({
  ...shiplightConfig(),
  reporter: [
    ["list"],
    [
      "shiplightai/reporter",
      {
        outputFolder: "my-report", // default: "shiplight-report"
        open: "on-failure", // "always" | "never" | "on-failure" (default)
      },
    ],
  ],
});

To regenerate the HTML without rerunning tests:

shiplight report                    # regenerate ./shiplight-report/index.html
shiplight report my-report --open   # regenerate and open in browser

In CI, shiplight report can upload test reports and their artifacts to Shiplight Cloud. See CI for setup.

Debugging

When a YAML test breaks, after a deploy, a UI change, or drift, there are two ways to fix it: let your agent do it, or step through it yourself in the visual debugger. Both work from the same evidence: the failure output, the YAML, the matching spec, and the live app in a browser when logs aren't enough.

Let the agent fix it: /shiplight fix

/shiplight fix reproduces a failing test, works out why from the evidence, and applies the smallest correct repair: a stale locator, a drifted assertion, expired auth, bad fixture data. Point it at a specific test or suite, or let it find the narrowest relevant target.

A suite broke after a deploy:

/shiplight fix the checkout suite is failing after the latest deploy

One test keeps flaking:

/shiplight fix tests/login.yaml times out on the 2FA step

In CI — the failure-triage pipeline can run fix automatically on red runs; see CI.

Passing tests are left alone unless they share the same broken source, and every fix is validated and rerun before it finishes.

The important part is what it won't do. If the app itself is broken, or its behavior now conflicts with the spec, it reports the mismatch instead of rewriting the test to paper over it, and it never deletes assertions or drops steps just to turn a test green. When intended behavior genuinely changed, it updates the spec first, then the test.

For authoring new tests, use /shiplight create-yaml-tests; to check a UI change when nothing is failing yet, use /shiplight verify.

Fix it yourself: the visual debugger

For complex UI flows, or when you'd rather drive the repair by hand, open the visual debugger, the human-in-the-loop path for hand-tuning tests. It opens a local web UI where you can browse tests, step through statements, inspect the browser, and edit YAML.

npx shiplight debug                                    # browse current directory
npx shiplight debug tests/                             # browse tests/ directory
npx shiplight debug tests/login.test.yaml              # open a specific test
npx shiplight debug tests/login.test.yaml --open       # open and launch browser

Create a New Test

Use --new to create a test file and start debugging immediately:

npx shiplight debug tests/checkout.test.yaml --new --url https://myapp.com/checkout

This creates the file with a starter template if it doesn't exist, then opens the debugger.

Options

Option	Description	Default
--port <n>	Server port	6174
--url <url>	Starting URL for new tests	—
--new	Create the test file if it doesn't exist	—
--open	Auto-open the debugger in your browser	—
--no-open	Don't auto-open the browser	default

The debugger auto-detects your playwright.config.ts and uses its settings (browser, baseURL, auth, etc.). If no config is found, it runs in standalone mode with a built-in browser sandbox.

CI/CD

Since npx shiplight test is a standard Playwright command, it works with any CI/CD provider: GitHub Actions, GitLab CI, CircleCI, Jenkins, etc. Just install dependencies and run:

npm ci && npx playwright install chromium && npx shiplight test

For Shiplight-hosted runners and the failure-triage pipeline, see CI & Auto-triage.