Local Testing with Shiplight CLI

View as Markdown

The Shiplight CLI (shiplightai) runs YAML tests locally — no cloud infrastructure required. It builds on top of Playwright with enhanced configuration, project scaffolding, and human-readable HTML reports. YAML tests run alongside your existing .test.ts files with no separate tooling.

The CLI includes an AI agent for natural language actions, self-healing locators, and VERIFY assertions — the same agent used in Shiplight Cloud execution.

Prerequisites

• Node.js >= 22
• AI API key — GOOGLE_API_KEY (Get key) or ANTHROPIC_API_KEY (Get key)

Store API keys and credentials in a .env file in your project root — the CLI auto-discovers it on startup. Make sure .env is in your .gitignore.

Quick Start

Invoke /create_e2e_tests in your coding agent (Claude Code, Cursor, or Codex) to get started. The agent scaffolds the project, installs dependencies, and configures API keys. From there, you can ask it to write YAML tests for your app.

Run tests with:

npx shiplight test

Project Structure

A typical project follows standard Playwright conventions. Shiplight adds .env for API keys and credentials.

my-tests/
├── playwright.config.ts
├── package.json
├── .env                            # API keys + credentials (gitignored)
├── .gitignore
│
├── tests/
│   ├── public-app/                 # No login needed
│   │   ├── search.test.yaml
│   │   └── filter.test.yaml
│   │
│   └── my-saas-app/               # Requires login
│       ├── auth.setup.ts           # Playwright login setup
│       ├── dashboard.test.yaml
│       └── settings.test.yaml

Run all tests:

npx shiplight test

Run one project:

npx shiplight test my-saas-app/

After the run completes, open shiplight-report/index.html to view the results. See Report Options for customization.

YAML Test Format

See YAML Test Format for a quick overview of statement types, actions, conditionals, loops, variables, and templates.

For the complete language specification and ready-to-run examples, see the examples repo.

Authentication (Optional)

If your app requires login, you can ask your AI coding agent to set up authentication for your test project — just describe your login flow and it will write the auth setup using one of the patterns below.

There are two approaches depending on whether all tests share the same account or individual tests need their own identity.

Shared Account (Most Common)

All tests share one authenticated session. This is the standard Playwright authentication pattern — a setup project logs in once, saves the browser state, and all tests reuse it.

// auth.setup.ts
import { test as setup } from "@playwright/test";

setup("login", async ({ page }) => {
  await page.goto("/login");
  await page.getByLabel("Email").fill(process.env.USERNAME!);
  await page.getByLabel("Password").fill(process.env.PASSWORD!);
  await page.getByRole("button", { name: "Sign in" }).click();
  await page.waitForURL("/dashboard");
  await page.context().storageState({ path: ".auth/default.json" });
});

// playwright.config.ts
export default defineConfig({
  ...shiplightConfig(),
  projects: [
    { name: "auth", testMatch: "auth.setup.ts" },
    {
      name: "default",
      dependencies: ["auth"],
      use: {
        baseURL: "https://staging.example.com",
        storageState: ".auth/default.json",
      },
    },
  ],
});

Tests don't need any account field — they're already authenticated via storageState. This is pure Playwright, no Shiplight-specific logic.

Multiple pre-defined accounts: If you have a small number of accounts (e.g., admin, viewer) but tests don't pick which one to use, parameterize auth.setup.ts with an environment variable and select the account at runtime:

TEST_ACCOUNT=admin npx shiplight test

This is still a shared account — just selected at runtime instead of hardcoded.

Per-Test Auth (Advanced)

When individual tests need their own identity (e.g., one test runs as admin, another as viewer in the same run), each test declares its auth script and optional args inline. This allows login customization per test.

1. Auth login function

Create an auth.login.ts that exports a login(args) function. It receives the args from the test config, performs the login flow, manages storageState caching and expiration, and returns the path to a storageState JSON file.

// auth.login.ts
import { chromium } from "@playwright/test";
import * as fs from "fs";
import * as path from "path";

export async function login(args: Record<string, unknown>): Promise<string> {
  const stateFile = path.join(".auth", `${args.username}.json`);

  // Return cached state if it exists (you control expiration logic here)
  if (fs.existsSync(stateFile)) return stateFile;

  // Perform login
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();

  await page.goto("/login");
  await page.getByLabel("Email").fill(args.username as string);
  await page.getByLabel("Password").fill(args.password as string);
  await page.getByRole("button", { name: "Sign in" }).click();
  await page.waitForURL("/dashboard");

  // Cache storageState (cookies + localStorage + IndexedDB)
  fs.mkdirSync(path.dirname(stateFile), { recursive: true });
  await context.storageState({ path: stateFile, indexedDB: true });
  await browser.close();

  return stateFile;
}

2. Use in tests

Each test declares auth (path to the login script) and optional args passed to the login function:

use:
  auth: ./auth.login.ts
  args:
    username: admin@example.com
    password: "{{ADMIN_PASSWORD}}"
goal: Admin can manage users
statements:
  - URL: /admin/users
  - VERIFY: User management page is visible

The args object is passed directly to the login function. This means your auth script can receive any fields your login flow needs (TOTP secrets, API tokens, org IDs, etc.). Tests that don't need args can omit the args field entirely.

The fixture uses the storage state path returned by the login() function to create the browser context, so the test runs in an already-logged-in state.

Tests without auth use the default context — if a setup project is configured, they get its storageState; otherwise they run unauthenticated.

Configuration

The shiplightConfig() helper in playwright.config.ts handles YAML transpilation and sets up the Shiplight reporter. It returns a partial Playwright config that you spread into defineConfig:

import { defineConfig, shiplightConfig } from "shiplightai";

export default defineConfig({
  ...shiplightConfig(),
  // Your Playwright config (testDir, projects, use, etc.)
});

Report

shiplightConfig() automatically enables the Shiplight HTML reporter — no extra configuration needed. After each run, open shiplight-report/index.html to view results with per-step screenshots, videos, and traces.

To customize the report output, override the reporter field in your config:

export default defineConfig({
  ...shiplightConfig(),
  reporter: [
    ["list"],
    [
      "shiplightai/reporter",
      {
        outputFolder: "my-report", // default: "shiplight-report"
        open: "on-failure", // "always" | "never" | "on-failure" (default)
      },
    ],
  ],
});

To regenerate the HTML without rerunning tests:

shiplight report                    # regenerate ./shiplight-report/index.html
shiplight report my-report --open   # regenerate and open in browser

Debugging

The shiplight debug command launches a visual debugger for YAML test files. It opens a local web UI where you can browse tests, step through statements, inspect the browser, and edit YAML — all in real time.

shiplight debug                                    # browse current directory
shiplight debug tests/                             # browse tests/ directory
shiplight debug tests/login.test.yaml              # open a specific test
shiplight debug tests/login.test.yaml --open       # open and launch browser

Create a New Test

Use --new to create a test file and start debugging immediately:

shiplight debug tests/checkout.test.yaml --new --url https://myapp.com/checkout

This creates the file with a starter template if it doesn't exist, then opens the debugger.

Options

Option	Description	Default
--port <n>	Server port	6174
--url <url>	Starting URL for new tests	—
--new	Create the test file if it doesn't exist	—
--open	Auto-open the debugger in your browser	—
--no-open	Don't auto-open the browser	default

The debugger auto-detects your playwright.config.ts and uses its settings (browser, baseURL, auth, etc.). If no config is found, it runs in standalone mode with a built-in browser sandbox.

CI/CD

Since shiplight test is a standard Playwright command, it works with any CI/CD provider — GitHub Actions, GitLab CI, CircleCI, Jenkins, etc. Just install dependencies and run:

npm ci && npx playwright install chromium && npx shiplight test