CLI Reference

View as Markdown

The shiplightai CLI is the entry point for every local Shiplight workflow: scaffolding a new project, running YAML and Playwright tests, debugging a test interactively, and regenerating HTML reports. It ships as an npm package (shiplightai) and is invoked through npx shiplight <command> from any project that has it installed.

This page covers how to install and keep the CLI up to date, and then every subcommand with its flags. If you are looking for the YAML test format, see YAML Test Format. If you are looking for browser automation features, see Browser Automation.

Installation

The CLI is meant to be installed as a project dependency inside the same directory as your tests — never globally. The easiest way to get started is the built-in scaffolder:

npx shiplightai@latest create ./my-tests
cd my-tests
cp .env.example .env                       # then edit .env and set an AI provider API key
npm install
npx playwright install chromium
npx shiplight debug tests/example.test.yaml --open

The scaffold includes tests/example.test.yaml — a self-contained starter test that visits shiplight.ai and verifies the Install Plugin link navigates to the docs. The final command opens it in the visual debugger so you can step through each statement live before writing your own tests.

The @latest prefix is only needed for the very first invocation, before anything is installed. Once the scaffolder has written a package.json with shiplightai as a dep and you have run npm install, every subsequent call can use plain npx shiplight <command> from inside the project.

Why not a global install

Global installs pin once and never auto-update, so the CLI and library silently drift apart from what's live on npm. shiplightai refuses to run when installed globally. If you have one from an earlier attempt, remove it:

npm uninstall -g shiplightai

Upgrading

Scaffolded projects pin shiplightai to the latest dist-tag in package.json:

{
  "dependencies": {
    "shiplightai": "latest"
  }
}

To pick up newer versions, run npm update from inside your project:

npm update shiplightai

This moves both package.json and package-lock.json forward to whatever the latest dist-tag currently points at on npm. CI keeps running the version captured in your lockfile until you commit an updated lockfile, so npm update locally is the mechanism for propagating new releases — there is no silent auto-update in CI.

Under the hood, the latest dist-tag is also what gives us a safe rollback lever: if a release turns out to be broken, we can move latest back to the previous version, and everyone running npm update on their next check picks up the good version.

Freshness warning

On every invocation, the CLI compares the running version against the latest published version on npm (cached locally for ~1 hour) and prints a yellow warning if you are behind:

⚠ shiplightai 0.1.49 is available (you have 0.1.47).
  Run: npm update shiplightai

The warning is a best-effort nag — completely silent on network failure, always skipped when CI=true is set in the environment (every major CI provider sets this automatically), and only prints for projects that actually have a package-lock.json (so the warning is always actionable by running npm update). The warning appears whenever the background check resolves — typically before your command output on a warm cache, or mid-run on a cold cache. Short-lived commands like --version may exit before the check completes, which is fine.

Node and Playwright requirements

• Node.js >= 22 — the CLI uses features that require this.
• @playwright/test — declared as a peer dependency of shiplightai (pinned to an exact version that the current CLI was built against). npm install picks it up automatically; you do not need to list it yourself.
• Chromium — only needed for browsers that use it. Install on demand with npx playwright install chromium.

Commands

All commands are invoked via npx shiplight <command> from inside a project that has shiplightai installed locally. You can also use them from npm scripts without the npx prefix, because npm run automatically puts node_modules/.bin on PATH.

Command	Purpose
shiplight create	Scaffold a new Shiplight test project
shiplight test	Run your YAML and Playwright test suite
shiplight debug	Launch the interactive visual debugger for a YAML test
shiplight report	Regenerate the HTML report from saved artifacts
shiplight transpile	Transpile YAML tests to Playwright .spec.ts files
shiplight --version	Print the installed CLI version
shiplight --help	Print top-level usage and list subcommands

shiplight create

Scaffolds a new Shiplight test project at the given path. See Installation above for the full first-run recipe. Used for manual terminal onboarding — the alternative to having a coding agent scaffold via /create_e2e_tests.

npx shiplightai@latest create <path> [options]

Generated files:

• package.json — declares shiplightai (pinned to latest) and dotenv as direct dependencies. @playwright/test is installed automatically as a peer dependency of shiplightai, so it does not need to be listed.
• playwright.config.ts — uses shiplightConfig() from shiplightai and wires up the Shiplight reporter
• .gitignore — excludes node_modules/, .env, test-results/, shiplight-report/, transpiled .yaml.spec.ts files
• .env.example — a starter file with commented examples for each supported AI provider
• tests/example.test.yaml — a self-contained starter test you can run immediately to verify your install works

Options:

Option	Description	Default
<path>	Directory where the project should be created (positional)	— (required)
--name <name>	package.json name field. Must be a valid lowercase npm package name (letters, digits, hyphens, underscores, dots).	basename of <path>
--help, -h	Print usage for this subcommand	—

The target directory must be empty (or not yet exist) — the scaffolder refuses to write into a non-empty directory to avoid silently overwriting files.

shiplight test

Runs your test suite. The CLI transpiles any YAML test files (*.test.yaml) to .yaml.spec.ts on the fly so they execute alongside your regular *.test.ts files in the same run. Any argument you pass after shiplight test is forwarded to Playwright, so the full Playwright CLI surface is available:

npx shiplight test                                  # run everything
npx shiplight test --headed                         # show the browser
npx shiplight test my-saas-app/                     # run one project
npx shiplight test tests/login.test.yaml            # run one file
npx shiplight test --grep "checkout"                # run by name pattern
npx shiplight test --workers 4                      # control parallelism

After the run completes, open shiplight-report/index.html to see results with per-step screenshots, videos, and traces. See shiplight report to regenerate or merge reports, and shiplight transpile if you want to inspect the generated .yaml.spec.ts code.

See Local Testing for the full project flow — environment variables, authentication, CI/CD integration, and project structure.

shiplight debug

Launches a visual debugger for a YAML test file. Opens a local web UI where you can browse tests, step through statements one at a time, inspect the page, edit YAML and re-run — all in real time against a live browser.

npx shiplight debug                                    # browse current directory
npx shiplight debug tests/                             # browse tests/ directory
npx shiplight debug tests/login.test.yaml              # open a specific test
npx shiplight debug tests/login.test.yaml --open       # also launch in browser

Options:

Option	Description	Default
--port <n>	Port for the debugger's local web server	6174
--url <url>	Starting URL for a newly created test	—
--new	Create the target file if it does not exist	—
--open	Auto-open the debugger in your default browser	—
--no-open	Do not auto-open the browser (default)	—

Creating a new test while debugging:

npx shiplight debug tests/checkout.test.yaml --new --url https://myapp.com/checkout --open

This creates tests/checkout.test.yaml from a starter template if it does not exist, then opens the debugger pointed at it.

The debugger auto-detects your playwright.config.ts and uses its settings (browser, baseURL, storage state, etc.). If no config is found, it runs in standalone mode with a built-in browser sandbox — useful for writing tests before your project has any Playwright setup.

The Shiplight VS Code extension embeds the same debugger directly in VS Code. When you install the extension and open a .test.yaml file, you get the full debugger UI in a side panel instead of a separate browser tab. The extension reuses your project-local CLI — there is no separate install step.

shiplight report

Regenerates or merges HTML reports from saved Playwright artifacts. Useful when you want to view a report without re-running tests, or to combine reports from parallel shards into one.

npx shiplight report                         # regenerate ./shiplight-report/index.html
npx shiplight report my-report               # regenerate a different folder
npx shiplight report my-report --open        # regenerate and open in browser
npx shiplight report --merge shard-1 shard-2 # merge multiple shard outputs

Options:

Option	Description
[folder]	Report output folder (positional, default ./shiplight-report)
--open	Open the generated report in your default browser
--merge	Merge multiple report directories into one combined report

The reporter is also wired up automatically by shiplightConfig() in your playwright.config.ts, so you do not normally need to run shiplight report after a regular shiplight test — the report is generated as part of the run. See Local Testing → Report for how to customize the reporter output during a run.

shiplight transpile

Transpiles YAML test files to standalone Playwright .yaml.spec.ts files. You normally do not need to run this manually — shiplight test transpiles on the fly — but it is useful when you want to inspect the generated code, check it into source control, or run it with a vanilla npx playwright test command.

npx shiplight transpile                            # transpile all **/*.test.yaml
npx shiplight transpile "tests/**/*.test.yaml"     # transpile a specific glob

Options:

Option	Description	Default
[glob]	Glob pattern of YAML test files to transpile (positional)	**/*.test.yaml

The generated .yaml.spec.ts files live alongside their .test.yaml sources. Add *.yaml.spec.ts to your .gitignore if you do not want to track them — the scaffolder does this by default.

Environment variables

Shiplight uses an explicit allowlist for env vars — only the variables in this table are forwarded from your shell (or .env file) into the agent's internal SDK config. Any env var that is not on this list is invisible to shiplightai by design: user test code cannot leak arbitrary environment state through the SDK.

All of these should be set in your project's .env file (auto-discovered by shiplightConfig()) or in your shell environment before running shiplight test.

LLM provider API keys

At least one is required. The first one set determines the default model (see Model selection).

Variable	Description	Default
GOOGLE_API_KEY	Google AI Studio key (Gemini models)	—
ANTHROPIC_API_KEY	Anthropic API key (Claude models)	—
OPENAI_API_KEY	OpenAI API key (GPT / o-series models)	—

Model selection

Variable	Description	Default
WEB_AGENT_MODEL	Override the web agent model (e.g. claude-sonnet-4-6). Supports provider:model prefix for azure:, bedrock:, vertex:.	Auto-detected from the first API key set
COMPUTER_USE_MODEL	Override the computer-use (CUA) model used for coordinate-based actions	Auto-detected

Custom OpenAI endpoint

Variable	Description	Default
OPENAI_BASE_URL	Custom base URL for OpenAI-compatible APIs (Ollama, vLLM, LiteLLM proxies). Used by both the regular LLM path and the computer-use path.	https://api.openai.com/v1

Vertex AI routing

Set one of the *_USE_VERTEXAI flags to route a provider through Google Vertex AI instead of the direct provider API. Either flag requires GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION to be set, and expects GOOGLE_APPLICATION_CREDENTIALS (or Application Default Credentials) to be available for google-auth-library.

Variable	Description	Default
ANTHROPIC_MODELS_USE_VERTEXAI	Route Anthropic (Claude) calls through Vertex AI	false
GOOGLE_GENAI_USE_VERTEXAI	Route Google (Gemini) calls through Vertex AI	false
GOOGLE_CLOUD_PROJECT	GCP project ID (required when either flag above is set)	—
GOOGLE_CLOUD_LOCATION	GCP region (e.g. us-central1, required when either flag set)	—

Tools

Variable	Description	Default
MAILGUN_API_KEY	Mailgun API key — required by the extract_email_content tool	—
MAILGUN_DOMAIN	Mailgun domain — required by extract_email_content	—

Logging

Variable	Description	Default
SDK_LOG_LEVEL	SDK log verbosity. One of debug, info, warn, error, silent (case-insensitive).	info

Adding a new env var

If you need Shiplight to honor a new environment variable, it must be added to the allowlist in apps/cli/src/fixture.ts — this is the fixture that forwards vars from process.env into the agent's SdkConfig.env. Setting an env var that is not on the allowlist will have no effect at runtime, even if sdk-core reads it internally.

Troubleshooting

shiplight: command not found

You are running shiplight as a bare command from your shell, but node_modules/.bin is not on your PATH. This is expected — the CLI is intentionally not designed for global installation. Use npx shiplight <command> instead, or run it through an npm script (npm run test, etc.), which automatically puts node_modules/.bin on PATH.

"cannot be run from a global install"

You hit the hard block on startup. The fix is in the error message itself:

npm uninstall -g shiplightai
cd <your-project>
npm install shiplightai
npx shiplight <command>

"A global shiplightai install was detected"

You have a project-local install (good) but also a leftover global install (not good). The coexisting global install can shadow your project's copy in some contexts. Remove it:

npm uninstall -g shiplightai

The warning is non-blocking — your command still ran — but the global install is a ticking time bomb for version skew, so clean it up.

The freshness warning keeps firing after npm update

If npm update shiplightai ran but the warning still shows the same numbers on the next invocation, check two things:

1. Your package.json version range. The scaffolder writes "shiplightai": "latest", which lets npm update cross any version boundary. If you manually pinned to something like "^0.1.11", npm update can only move you within that range and may silently do nothing when a newer minor ships. Change it to "latest" or widen the range.
2. The local version cache. The CLI caches the latest-version lookup for ~1 hour in ~/.shiplight/version-check.json. If the cache is fresh but wrong, rm ~/.shiplight/version-check.json and try again.

Running in CI and want the warning silenced

The warning is automatically suppressed when the CI environment variable is set to a truthy value — all major CI providers (GitHub Actions, GitLab, CircleCI, Jenkins, etc.) do this for you. If you are running in an environment that does not set CI but still want the warning silenced, set it yourself:

CI=true npx shiplight test

The scaffolded project has deprecation warnings on npm install

Some warnings come from transitive dependencies (for example node-domexception) and are outside the CLI's direct control. These are non-fatal.

Related pages

• Local Testing — project structure, authentication, environment variables, CI/CD
• YAML Test Format — statement types, actions, locators, variables, templates
• Browser Automation — persistent profiles, mobile emulation, extension testing
• VS Code Extension — embedding the debugger into VS Code
• Report to Cloud — uploading reports for sharing
• Sync to Cloud — syncing local tests with Shiplight Cloud