Writing Test Flows

View as Markdown

Shiplight tests are written in YAML using natural language. AI agents author and enrich the test flows, while the natural language format keeps tests readable for human review and modification.

No lock-in: Test flows can be run locally with Playwright using shiplightai. The YAML format is an authoring layer — what actually runs is standard Playwright with an AI agent on top. You can eject at any time.

Basic Structure

goal: Description of what this test verifies
url: https://your-app.com/starting-page

statements:
  - Step described in natural language
  - Another step
  - "VERIFY: Expected outcome"

teardown:
  - Clean up step

Field	Required	Description
goal	Yes	Test description (used as the Playwright test name)
url	Yes	Starting URL to navigate to
statements	Yes	List of test steps
teardown	No	Steps that always run after the test (like finally)

Basic Test

Every line is a natural language instruction. The web agent resolves each one at runtime by looking at the page and performing the right action.

goal: Verify user can create a new project
url: https://app.example.com/projects
statements:
  - Click the "New Project" button
  - Enter "My Test Project" in the project name field
  - Select "Public" from the visibility dropdown
  - Click "Create"
  - "VERIFY: Project page shows title 'My Test Project'"
teardown:
  - Delete the created project

Enriched Test

After exploring the UI with the browser automation tools (get_dom, act, etc.), the coding agent enriches natural language steps with locators for deterministic, fast replay:

goal: Verify user can create a new project
url: https://app.example.com/projects
statements:
  - STEP: Create project
    statements:
      - description: Click the New Project button
        action_entity:
          action_data:
            action_name: click
          locator: "getByRole('button', { name: 'New Project' })"
      - description: Enter project name
        action_entity:
          action_data:
            action_name: input_text
            kwargs:
              text: "My Test Project"
          locator: "getByRole('textbox', { name: 'Project name' })"
      - description: Click Create
        action_entity:
          action_data:
            action_name: click
          locator: "getByRole('button', { name: 'Create' })"
  - "VERIFY: Project page shows title 'My Test Project'"
teardown:
  - Delete the created project

Natural language statements (~10-15s each): The web agent reads the page and figures out what to do. Action statements with locator (~1s each): Replay deterministically without AI. VERIFY statements: Always use the web agent — just provide a clear assertion in natural language.

Locators Are a Cache

Locators are a performance cache, not a hard dependency. When the UI changes and a locator becomes stale, Shiplight's agentic layer auto-heals by falling back to the natural language description to find the right element.

However, when a locator is permanently changed (e.g., a button was renamed or moved), the cached locator will fail on every run. When running on the Shiplight cloud, the platform self-updates the cached locator after a successful self-heal — so future runs replay at full speed again without manual intervention. This self-adjusting behavior is a key benefit of running tests on the Shiplight platform.

Statement Types

Type	Syntax	Description
Natural language	- Click the login button	Web agent resolves at runtime
Action with locator	- description: ... + action_entity: ...	Deterministic replay
Verify	- "VERIFY: page shows welcome message"	AI-powered assertion
Step group	- STEP: Login + statements: [...]	Group related actions
Conditional	- IF: cookie banner is visible + THEN: [...]	Conditional execution
Loop	- WHILE: more items to load + DO: [...]	Repeat until condition

VERIFY

Asserts a condition using AI. Must be a quoted string prefixed with VERIFY:.

statements:
  - "VERIFY: The success message is displayed"
  - "VERIFY: User is redirected to the dashboard"

ACTION (with action entity)

Deterministic actions with explicit locators. These replay fast (~1s) without AI.

statements:
  - description: Click the Submit button
    action_entity:
      action_description: Click the Submit button
      locator: "getByRole('button', { name: 'Submit' })"
      action_data:
        action_name: click
        kwargs: {}

  - description: Type email address
    action_entity:
      action_description: Type email address
      locator: "getByLabel('Email')"
      action_data:
        action_name: input_text
        kwargs:
          text: "user@example.com"

  - description: Press Enter
    action_entity:
      action_data:
        action_name: press
        kwargs:
          keys: Enter

STEP (grouping)

Groups related statements under a label.

statements:
  - STEP: Fill in the registration form
    statements:
      - Type "John" in the first name field
      - Type "Doe" in the last name field
      - Type "john@example.com" in the email field

Locators

Action entities can specify element locators in two ways:

# Playwright locator (preferred)
locator: "getByRole('button', { name: 'Submit' })"

# XPath
xpath: "//button[@id='submit']"

If both are present, locator takes priority. If neither is present, the AI agent resolves the element from the action description.

Frames

For elements inside iframes:

action_entity:
  frame_path:
    - "iframe#main"
  locator: "getByText('Hello')"
  action_data:
    action_name: click
    kwargs: {}

Conditional Logic

Handle optional UI elements with IF/ELSE:

statements:
  - IF: cookie consent dialog is visible
    THEN:
      - Click "Accept All"
  - IF: user is logged in
    THEN:
      - Click the logout button
    ELSE:
      - Click the login button
      - Enter credentials and submit

Conditions are evaluated by the web agent at runtime using the current page state. JavaScript conditions are also supported with the js: prefix:

- IF: "js: page.url().includes('/login')"
  THEN:
    - Enter credentials and log in

Loops

Repeat actions until a condition is met with WHILE:

statements:
  - WHILE: "Load More" button is visible
    DO:
      - Click the "Load More" button
      - Wait for new items to appear
    timeout_ms: 30000
  - "VERIFY: all items are loaded"

JavaScript conditions work in loops too:

- WHILE: "js: document.querySelectorAll('.item').length < 10"
  DO:
    - Click "Load More"

Supported Actions

Actions used in action_entity.action_data.action_name:

Action	kwargs	Description
click	—	Click an element
double_click	—	Double-click an element
right_click	—	Right-click an element
hover	—	Hover over an element
input_text	text	Type text into an input
clear_input	—	Clear an input field
press	keys	Press a keyboard key (e.g., Enter, Tab)
send_keys_on_element	keys	Press a key on a specific element
select_dropdown_option	text	Select a dropdown option by text
scroll	down, num_pages	Scroll the page
scroll_to_text	text	Scroll to text on the page
go_to_url	url, new_tab	Navigate to a URL
go_back	—	Browser back
reload_page	—	Reload the page
wait	seconds	Wait for a duration
wait_for_page_ready	—	Wait for page load
verify	statement or code	Assert a condition (AI or JS)
js_code	code	Run inline JavaScript
function	functionName, parameterNames, parameterValues	Call a function
switch_tab	tab_index	Switch browser tab
close_tab	—	Close current tab
upload_file	file_path	Upload a file
save_variable	name, value	Save a variable for later use

Extensions

Custom Test Name

Override the Playwright test name (defaults to goal):

name: Login with valid credentials
goal: Verify login flow works
url: https://example.com
statements:
  - ...

Tags

Add Playwright tags for filtering with --grep:

tags:
  - smoke
  - auth
goal: Login test
url: https://example.com
statements:
  - ...

Run: npx playwright test --grep @smoke

Playwright Fixtures

Pass options to test.use():

use:
  viewport:
    width: 375
    height: 812
  locale: fr-FR
goal: Mobile French layout
url: https://example.com
statements:
  - ...

Variables

Use {{VAR_NAME}} to reference environment variables. At transpile time, these become process.env.VAR_NAME references in the generated code.

statements:
  - description: Type username
    action_entity:
      locator: "getByLabel('Username')"
      action_data:
        action_name: input_text
        kwargs:
          text: "{{TEST_USER}}"

Set variables when running:

TEST_USER=admin TEST_PASS=secret npx playwright test

Templates

Extract reusable flows into template files and include them with template:.

Template file (templates/login.yaml):

params:
  - username
  - password

statements:
  - description: Enter username
    action_entity:
      locator: "getByLabel('Username')"
      action_data:
        action_name: input_text
        kwargs:
          text: "{{username}}"
  - description: Enter password
    action_entity:
      locator: "getByLabel('Password')"
      action_data:
        action_name: input_text
        kwargs:
          text: "{{password}}"
  - description: Click login
    action_entity:
      locator: "getByRole('button', { name: 'Log in' })"
      action_data:
        action_name: click
        kwargs: {}

Using the template:

goal: Purchase flow
url: https://example.com

statements:
  - template: ../templates/login.yaml
    params:
      username: "{{TEST_USER}}"
      password: "{{TEST_PASS}}"
  - Navigate to the checkout page
  - "VERIFY: Order summary is displayed"

Template params ({{username}}) are substituted at transpile time. Environment variables ({{TEST_USER}}) pass through to the generated code for runtime resolution.

Templates can be nested (max depth: 5) and circular references are detected.

Custom Functions

Call TypeScript functions from YAML using the function action with file#export syntax:

statements:
  - description: Seed test data
    action_entity:
      action_data:
        action_name: function
        kwargs:
          functionName: "../helpers/seed.ts#createTestUser"
          parameterNames:
            - page
            - email
          parameterValues:
            - page
            - "test@example.com"

This generates:

import { createTestUser } from "../helpers/seed";
// ...
await createTestUser(page, "test@example.com");

The Enrichment Workflow

1. Draft — The agent writes tests in natural language
2. Explore — The agent uses get_dom and act to walk through the UI
3. Collect — The agent uses get_locator to capture element locators
4. Enrich — The agent replaces natural language steps with action_entity + locator
5. Result — Tests run 10x faster with deterministic replay

Natural language and enriched statements can be mixed in the same test. The agent starts with all natural language, then selectively enriches the most-used flows.