Local Testing with Shiplight CLI

View as Markdown

The Shiplight CLI (shiplightai) runs YAML E2E tests locally — end-to-end browser tests authored in YAML instead of Playwright code. Coding agents create and maintain these tests, while humans can review the readable YAML, run the tests locally, and hand-tune complex UI flows in the debugger. YAML E2E tests run alongside your existing .test.ts files with no separate tooling.

The CLI includes an AI agent for natural language actions, self-healing locators, and VERIFY assertions — the same agent used in Shiplight Cloud execution.

Let your agent set this up

You rarely need to scaffold or wire this by hand. Run /shiplight init to scaffold the project and /shiplight create-yaml-tests to author tests — your coding agent handles dependencies, API keys, playwright.config.ts, and the first tests. This page documents what those produce so you can review, run, and hand-tune the results. See the full /shiplight skill.

On this section

• Authentication — shared-account and per-test login patterns
• Configuration — shiplightConfig() and report options
• Debugging — the visual debugger for hand-tuning tests
• CLI Reference — every shiplight command and flag

Prerequisites

• Node.js >= 22
• AI API key — GOOGLE_API_KEY (Get key) or ANTHROPIC_API_KEY (Get key)

Store API keys and credentials in a .env file in your project root — the CLI auto-discovers it on startup. Make sure .env is in your .gitignore.

Quick Start

Invoke /shiplight create-yaml-tests in your coding agent (Claude Code, Cursor, or Codex) to get started. The agent scaffolds the project, installs dependencies, configures API keys, and writes the first YAML E2E tests for your app.

Run tests with:

npx shiplight test

Project Structure

A typical project follows standard Playwright conventions. Shiplight adds .env for API keys and credentials.

my-tests/
├── playwright.config.ts
├── package.json
├── .env                            # API keys + credentials (gitignored)
├── .gitignore
│
├── tests/
│   ├── public-app/                 # No login needed
│   │   ├── search.test.yaml
│   │   └── filter.test.yaml
│   │
│   └── my-saas-app/               # Requires login
│       ├── auth.setup.ts           # Playwright login setup
│       ├── dashboard.test.yaml
│       └── settings.test.yaml

Run all tests:

npx shiplight test

Run one project:

npx shiplight test my-saas-app/

After the run completes, open shiplight-report/index.html to view the results. See Configuration → Report for customization.

YAML E2E Test Format

See YAML E2E Test Format for a quick overview of statement types, actions, conditionals, loops, variables, and templates.

For the complete language specification and ready-to-run examples, see the examples repo.

CI/CD

Since npx shiplight test is a standard Playwright command, it works with any CI/CD provider — GitHub Actions, GitLab CI, CircleCI, Jenkins, etc. Just install dependencies and run:

npm ci && npx playwright install chromium && npx shiplight test

For Shiplight-hosted runners and the failure-triage pipeline, see CI & Auto-triage.