Code Steps

View as Markdown

A code step — a js: block with a description: label — lets you run arbitrary JavaScript directly inside a YAML test. It executes in the same Node.js context as Playwright, giving you full access to page, browser APIs, HTTP requests, and the test variable store.

Use it when a natural language step isn't precise enough — calling an API, intercepting network requests, seeding browser state, capturing values mid-test, or writing exact assertions.

Table of Contents

1. Syntax
2. Available Context
3. Failing a Step
4. Common Use Cases
   • Call APIs
   • Page Operations
   • Store Values for Later Steps
   • Custom Assertions
   • Verify a Downloaded File
5. Code step vs VERIFY js: vs structured actions
6. Limitations

Syntax

A code step is a js: field with a description: label that says what the code does. The description: is optional but recommended — it's the human-readable label shown in reports.

Single-line:

statements:
  - description: Wait for the network to be idle
    js: "await page.waitForLoadState('networkidle')"

Multi-line (block scalar):

statements:
  - description: Mock the users API
    js: |
      await page.route('**/api/users', (route) => route.fulfill({
        status: 200,
        body: JSON.stringify([{ name: 'Alice' }]),
      }));
      testContext.apiMocked = true;

Quoting rules

Use double quotes for single-line code that doesn't contain double quotes. Use the block scalar (|) for multi-line code or any code containing special YAML characters ({, }, :, #).

Available Context

The code runs in Node.js (not the browser). The following variables are available without any imports:

Variable	Type	Description
page	Playwright Page	The current browser page
context	Playwright BrowserContext	The browser context (cookies, permissions, etc.)
expect	Playwright assertions	For writing assertions
request	Playwright APIRequestContext	For making HTTP requests
testContext	Shiplight variable store	Read and write test variables. Aliases: $, ctx
agent	Shiplight WebAgent	AI-powered actions — assertions, natural language step execution
console	Captured console	Logs appear in test output

Async/await is fully supported. All code runs inside an async context, so you can freely use await.

To run code inside the browser (accessing window, document, localStorage, etc.), use page.evaluate():

- description: Set a feature flag in localStorage
  js: |
    await page.evaluate(() => {
      localStorage.setItem('featureFlag', 'true');
    });

Working with testContext

testContext (also $ and ctx) is a proxy to the test's variable store. Values set here are available in all subsequent steps, including natural language steps.

// Set a variable
testContext.orderId = "12345";
$.theme = "dark";

// Read a variable
const id = testContext.orderId;

// Set a sensitive value (masked in logs)
testContext.set("apiToken", "secret-value", true);

// Read explicitly
const token = testContext.get("apiToken");

Using agent

The agent object gives you access to Shiplight's AI capabilities from inside a code step.

agent.assert(page, statement) — AI-powered assertion. The agent looks at the current page and verifies whether the statement is true.

- description: Assert the cart shows 3 items
  js: |
    await agent.assert(page, 'The shopping cart shows 3 items');

agent.execute(page, statement) — Execute a natural language instruction. The agent reads the page and performs the described action, just like a natural language step.

- description: Fill and submit the registration form
  js: |
    await agent.execute(page, 'Fill out the registration form with test data');
    await agent.execute(page, 'Click the Submit button');

agent.extract(page, description, variableName) — Extract a value from the page using AI and store it in testContext.

- description: Extract the order confirmation number
  js: |
    await agent.extract(page, 'the order confirmation number', 'orderNumber');
    console.log('Order:', testContext.orderNumber);

agent.getRecentDownloadedFilePath() — Get the path of the most recently downloaded file.

- description: Capture the downloaded file path
  js: |
    await agent.waitForDownloadComplete(page, 10);
    const filePath = agent.getRecentDownloadedFilePath();
    console.log('Downloaded:', filePath);

Failing a Step

The return value of the code is ignored. To fail a step, either throw an error or use expect():

// Fail via expect (throws if assertion doesn't hold):
const title = await page.title();
expect(title).toContain("Dashboard");

// Fail via throw:
const count = await page.locator(".item").count();
if (count === 0) {
  throw new Error("Expected at least one item, found none");
}

Common Use Cases

Call APIs

Use request to call HTTP endpoints directly from a test step — useful for seeding test data, triggering backend actions, or checking state that isn't visible in the UI.

statements:
  - description: Create an order via the API
    js: |
      const response = await request.post('https://api.example.com/orders', {
        headers: { Authorization: `Bearer ${testContext.apiToken}` },
        data: { productId: 'prod_123', quantity: 2 },
      });
      expect(response.ok()).toBeTruthy();
      const order = await response.json();
      testContext.orderId = order.id;
  - URL: "https://app.example.com/orders/{{orderId}}"
  - VERIFY: Order details page is displayed

request uses the same authentication cookies as the browser session, so it works for authenticated API calls without any extra setup.

Page Operations

Interact directly with the browser — mock network requests, seed storage, inject cookies, or wait for specific states.

Mock an API response:

statements:
  - description: Mock the products API
    js: |
      await page.route('**/api/products', (route) => route.fulfill({
        status: 200,
        contentType: 'application/json',
        body: JSON.stringify([{ id: 1, name: 'Widget', price: 9.99 }]),
      }));
  - URL: https://app.example.com/shop
  - VERIFY: Widget is listed with price $9.99

Seed browser storage before a page load:

statements:
  - URL: https://app.example.com
  - description: Mark onboarding complete in localStorage
    js: |
      await page.evaluate(() => {
        localStorage.setItem('onboarding_complete', 'true');
      });
  - URL: https://app.example.com/dashboard
  - VERIFY: Onboarding banner is not shown

Inject a session cookie:

- description: Inject a session cookie
  js: |
    await context.addCookies([{
      name: 'session',
      value: 'abc123',
      domain: 'app.example.com',
      path: '/',
    }]);

Wait for a condition:

- description: Wait for the network to be idle
  js: "await page.waitForLoadState('networkidle')"

- description: Wait until at least 10 items are loaded
  js: |
    await page.waitForFunction(() => {
      return document.querySelectorAll('.item').length >= 10;
    }, { timeout: 10000 });

Store Values for Later Steps

Read something from the page and save it to testContext so later steps can reference it.

statements:
  - intent: Submit the order form
  - description: Capture the order number
    js: |
      const orderNumber = await page.locator('[data-testid="order-number"]').textContent();
      testContext.orderNumber = orderNumber?.trim();
  - URL: https://app.example.com/order-history
  - VERIFY: "Order {{orderNumber}} appears in the history"

Values stored in testContext are available in all subsequent steps — both code steps and natural language steps — for the duration of the test run.

Custom Assertions

Write exact, programmatic checks for cases where VERIFY: isn't precise enough — counts, numeric ranges, string patterns, or multi-element checks.

- description: Assert the table has 5 rows
  js: |
    const rows = await page.locator('table tbody tr').count();
    expect(rows).toBe(5);

- description: Assert the cart total is within range
  js: |
    const total = await page.locator('[data-testid="cart-total"]').textContent();
    const amount = parseFloat(total!.replace('$', ''));
    expect(amount).toBeGreaterThan(0);
    expect(amount).toBeLessThan(1000);

Use throw when expect() doesn't fit:

- description: Assert the order status is Shipped or Delivered
  js: |
    const status = await page.locator('.order-status').textContent();
    if (!['Shipped', 'Delivered'].includes(status?.trim() ?? '')) {
      throw new Error(`Unexpected order status: ${status}`);
    }

Verify a Downloaded File

Shiplight automatically saves files to disk when a download is triggered. Use agent methods to wait for the download to complete and verify the result.

statements:
  - intent: Click the Export CSV button
  - description: Verify a CSV file was downloaded
    js: |
      // Wait up to 10 seconds for the download to finish
      await agent.waitForDownloadComplete(page, 10);

      // Get the path of the most recently downloaded file
      const filePath = agent.getRecentDownloadedFilePath();
      expect(filePath).toBeTruthy();
      expect(filePath).toContain('.csv');
      console.log('Downloaded file:', filePath);

Verifying file contents

The code step sandbox does not have access to the filesystem, so you cannot read file contents directly. To verify what's inside a downloaded file, use a custom function — custom functions run with full Node.js access, including fs.

Code step vs VERIFY js: vs structured actions

All three let you write Playwright code, but they serve different purposes:

	description: + js: (code step)	VERIFY: + js:	intent: + action:/locator:
Self-healing	No	Yes — falls back to AI vision	Yes — agent re-derives from intent:
Purpose	Setup, utilities, mocking, custom assertions	Assertions with an AI fallback	UI interactions with a fast-replay cache
Failure behavior	Throws → test fails immediately	Throws → agent evaluates VERIFY: via AI	Locator fails → agent re-derives from intent:
Best for	Network setup, storage seeding, computed checks	Checking visible outcomes	Clicks, typing, keyboard, hover

Use a code step (description: + js:) when the operation is deterministic and doesn't need recovery — setting up mocks, seeding data, computing derived values, or asserting precise conditions.

Use VERIFY: + js: when you have a preferred assertion but want the AI to verify the outcome in natural language if the code path fails.

Use a structured action: when the step interacts with the UI and you want AI-powered self-healing if the page changes.

Limitations

• No dynamic imports. You cannot use import() inside a code step. For reusable logic, use custom functions instead.
• Node.js context only. Code runs on the Node.js side, not in the browser. Use page.evaluate() to access browser globals like window, document, or localStorage.
• No return value. The return value of the code block is discarded. Use testContext to pass values between steps.