Local Testing

View as Markdown

Run YAML test flows locally with Playwright — no cloud infrastructure required. Write tests in natural language, run them with npx playwright test.

Tests use the same AI-powered web agent as cloud execution: natural language actions, self-healing locators, and VERIFY assertions. Compatible with your existing Playwright setup — YAML tests run alongside .test.ts files with no separate tooling.

Prerequisites

• Node.js >= 22
• AI API key — at least one of:
  • ANTHROPIC_API_KEY (Get key)
  • GOOGLE_API_KEY (Get key)

Quick Start

1. Install

npm install -D shiplightai @shiplightai/sdk-pro shiplight-types @playwright/test

2. Configure

In your playwright.config.ts:

import { defineConfig } from "@playwright/test";
import { shiplightConfig } from "shiplightai";

export default defineConfig({
  ...shiplightConfig(),

  testDir: "./tests",
  use: {
    headless: true,
    viewport: { width: 1280, height: 720 },
  },
});

3. Set up API keys

At least one AI API key is required. You can either export it directly or use a .env file.

Option A: Environment variable

export ANTHROPIC_API_KEY=sk-ant-...
# or export GOOGLE_API_KEY=...

Option B: .env file (in your test directory or project root)

ANTHROPIC_API_KEY=sk-ant-...
# GOOGLE_API_KEY=...

shiplightConfig() auto-discovers .env files by walking up the directory tree — no manual dotenv setup needed.

The AI model is auto-detected from your API key (ANTHROPIC_API_KEY → claude-haiku-4-5, GOOGLE_API_KEY → gemini-2.5-pro). Set WEB_AGENT_MODEL to override.

4. Write a YAML test

Create tests/login.test.yaml:

goal: Verify user can log in
url: https://example.com/login

statements:
  - Click on the username field and type "testuser"
  - Click on the password field and type "secret123"
  - Click the Login button
  - "VERIFY: Dashboard page is visible"

5. Run

npx playwright test

Playwright discovers both *.test.ts and *.test.yaml files. YAML files are transparently transpiled to .yaml.spec.ts files next to the source.

6. Gitignore generated files

Add to your .gitignore:

*.yaml.spec.ts
.env

How It Works

shiplightConfig() runs during Playwright config loading and:

1. Walks up from scanDir to the project root looking for .env files and loads them (closer files take precedence)
2. Scans for **/*.test.yaml files
3. Transpiles each to a *.yaml.spec.ts file next to the source

Playwright then discovers the generated .yaml.spec.ts files through its default testMatch pattern. Each generated test uses a custom test fixture that extends Playwright's test with an agent (WebAgent) instance — so your YAML steps get AI-powered execution automatically.

Project Structure

A typical project follows standard Playwright conventions. Shiplight adds .env for API keys and shiplight.config.json for per-project login credentials (only needed if the app requires authentication).

my-tests/
├── playwright.config.ts
├── package.json
├── .env                            # API keys (gitignored)
├── .gitignore
│
├── airbnb/                         # Project 1: public site, no login
│   ├── search.test.yaml
│   └── filter.test.yaml
│
├── my-saas-app/                    # Project 2: requires login
│   ├── shiplight.config.json       # {"url":"...","username":"...","password":"..."}
│   ├── dashboard.test.yaml
│   └── settings.test.yaml
│
└── admin-portal/                   # Project 3: different app, different login
    ├── shiplight.config.json
    └── users.test.yaml

Run all tests:

npx playwright test

Run one project:

npx playwright test my-saas-app/

YAML Test Format

See Writing Test Flows for the full YAML reference (statement types, conditionals, loops, action entities). Here's a quick summary:

goal: Description of what this test verifies
url: https://your-app.com/starting-page

statements:
  - Step described in natural language # AI-resolved (~10-15s)
  - "VERIFY: Expected outcome" # AI assertion
  - description: Click the Submit button # Deterministic replay (~1s)
    action_entity:
      locator: "getByRole('button', { name: 'Submit' })"
      action_data:
        action_name: click
        kwargs: {}

teardown:
  - Clean up step # Always runs (like finally)

Extensions

name: Custom test name # Override Playwright test name (defaults to goal)
tags: # Filter with npx playwright test --grep @smoke
  - smoke
  - auth
use: # Passed to test.use() — Playwright fixtures
  viewport:
    width: 375
    height: 812
  locale: fr-FR

Variables

Use to reference environment variables at runtime:

statements:
  - description: Type username
    action_entity:
      locator: "getByLabel('Username')"
      action_data:
        action_name: input_text
        kwargs:
          text: "{{TEST_USER}}"

Templates

Extract reusable flows into template files:

# templates/login.yaml
params:
  - username
  - password
statements:
  - description: Enter username
    action_entity:
      locator: "getByLabel('Username')"
      action_data:
        action_name: input_text
        kwargs:
          text: "{{username}}"

# tests/checkout.test.yaml
goal: Purchase flow
url: https://example.com
statements:
  - template: ../templates/login.yaml
    params:
      username: "{{TEST_USER}}"
      password: "{{TEST_PASS}}"
  - Navigate to the checkout page
  - "VERIFY: Order summary is displayed"

Custom Functions

Call TypeScript functions from YAML:

statements:
  - description: Seed test data
    action_entity:
      action_data:
        action_name: function
        kwargs:
          functionName: "../helpers/seed.ts#createTestUser"
          parameterNames:
            - page
            - email
          parameterValues:
            - page
            - "test@example.com"

Authentication (Optional)

If your app requires login, use Playwright's globalSetup to authenticate once and share the session across all test workers.

1. Create shiplight.config.json

Place in your test subdirectory:

{
  "url": "https://your-app.com",
  "username": "testuser@example.com",
  "password": "your-password",
  "totp_secret": "JBSWY3DPEHPK3PXP"
}

The totp_secret field is optional — only needed for 2FA.

2. Create global-setup.ts

This runs once before all tests. It uses WebAgent.loginPage() for AI-driven login — no fragile selectors that break when your login UI changes.

import * as path from "path";
import { chromium, type FullConfig } from "@playwright/test";
import { readFile, mkdir } from "fs/promises";
import { LoginType } from "shiplight-types";
import { resolveLoginConfig } from "shiplightai";

const AUTH_DIR = ".auth";
const STORAGE_STATE_PATH = `${AUTH_DIR}/storage-state.json`;

async function loadStorageState(): Promise<any | undefined> {
  try {
    const data = await readFile(STORAGE_STATE_PATH, "utf-8");
    return JSON.parse(data);
  } catch {
    return undefined;
  }
}

async function globalSetup(config: FullConfig) {
  const baseURL = config.projects[0]?.use?.baseURL || process.env.PLAYWRIGHT_BASE_URL || "https://your-app.com";

  const testDir = path.resolve(__dirname);
  const loginConfig = resolveLoginConfig(testDir);

  if (!loginConfig) {
    console.log("[global-setup] No login credentials found.");
    return;
  }

  const { username, password, loginUrl, totpSecret } = loginConfig;

  const { WebAgent, createAgentContext, configureSdk, VariableStore } = await import("@shiplightai/sdk-pro");

  configureSdk({
    env: {
      GOOGLE_API_KEY: process.env.GOOGLE_API_KEY ?? "",
      ANTHROPIC_API_KEY: process.env.ANTHROPIC_API_KEY ?? "",
    },
  });

  const { resolveModelFromEnv } = await import("shiplight-types");
  const model = resolveModelFromEnv();
  if (!model) {
    console.log("[global-setup] No AI model configured.");
    return;
  }

  const agent = new WebAgent(
    createAgentContext({
      model,
      variableStore: new VariableStore(),
    }),
  );

  const browser = await chromium.launch();
  try {
    const storageState = await loadStorageState();
    const context = await browser.newContext({
      baseURL,
      ...(storageState && { storageState }),
    });
    const page = await context.newPage();

    const absoluteLoginUrl = loginUrl ? (loginUrl.startsWith("http") ? loginUrl : `${baseURL}${loginUrl}`) : baseURL;

    const result = await agent.loginPage(page, {
      site_url: absoluteLoginUrl,
      num_verification_exprs: 0,
      account: {
        type: LoginType.PASSWORD,
        username,
        password,
        ...(totpSecret && {
          two_factor_auth_config: { type: "totp", data: totpSecret },
        }),
      },
    });

    if (!result.success) {
      throw new Error("[global-setup] Login failed.");
    }

    await mkdir(AUTH_DIR, { recursive: true });
    await context.storageState({ path: STORAGE_STATE_PATH });
    console.log("[global-setup] Auth state saved.");
  } finally {
    await browser.close();
  }
}

export default globalSetup;

3. Add globalSetup to config

export default defineConfig({
  ...shiplightConfig(),

  globalSetup: "./global-setup.ts",
  use: {
    storageState: ".auth/storage-state.json",
  },
});

Credential resolution

Login credentials are resolved in this order (first match wins):

1. Environment variables: SHIPLIGHT_LOGIN_EMAIL + SHIPLIGHT_LOGIN_PASSWORD
2. Config file: shiplight.config.json (discovered by walking up from test directory)

SHIPLIGHT_LOGIN_URL and SHIPLIGHT_LOGIN_TOTP_SECRET always override config file values when set.

Configuration Options

shiplightConfig({
  // Directory to scan for .test.yaml files (default: process.cwd())
  scanDir: "./tests",

  // API key for cloud features (optional)
  apiKey: process.env.SHIPLIGHT_API_KEY,

  // Auto-discover .env files by walking up (default: true)
  // Set to false for CI where env vars are injected externally
  dotenv: false,
});

Agent Fixture

The package exports a custom test that extends Playwright's test with an agent fixture. It works exactly like Playwright's test in every other way. expect is re-exported from Playwright unchanged.

import { test, expect } from "shiplightai/fixture";

test("custom test with agent", async ({ page, agent }) => {
  await page.goto("https://example.com");
  await agent.run(page, "Click the login button", "step-1");
  await agent.assert(page, "User is on the dashboard", "step-2");
});

You can mix hand-written .test.ts files with .test.yaml files in the same project.

Environment Variables

Variable	Description	Default
ANTHROPIC_API_KEY	Anthropic API key (for Claude models)	—
GOOGLE_API_KEY	Google AI API key (for Gemini models)	—
WEB_AGENT_MODEL	AI model override	Auto-detected from API key
SHIPLIGHT_LOGIN_EMAIL	Login email (overrides shiplight.config.json)	—
SHIPLIGHT_LOGIN_PASSWORD	Login password (overrides shiplight.config.json)	—
SHIPLIGHT_LOGIN_URL	Login page URL override	—
SHIPLIGHT_LOGIN_TOTP_SECRET	TOTP secret for 2FA	—
PLAYWRIGHT_STARTING_URL	Override starting URL for all tests	—

At least one AI API key is required. The model is auto-detected: ANTHROPIC_API_KEY defaults to claude-haiku-4-5, GOOGLE_API_KEY defaults to gemini-2.5-pro.

Local vs. Cloud Execution

Aspect	Local (shiplightai)	Cloud (run_test_case)
Infrastructure	Your machine	Shiplight cloud runners
Setup	npm install + API key	No setup — runs on Shiplight
Locator self-update	No (locators are static)	Yes (auto-updates after self-heal)
Parallel execution	Via Playwright --workers	Built-in parallelism
Best for	Local dev, CI pipelines, offline runs	Continuous regression, scheduled runs

Both modes support the same YAML test flow format and the same statement types (natural language, enriched actions, VERIFY, IF/ELSE, WHILE loops, templates, variables, and custom functions).