Cloud Testing

View as Markdown

LEGACY — Shiplight Cloud v1

This page documents Shiplight Cloud v1 (app.shiplight.ai), which is no longer the current product. For the current cloud platform, see Shiplight Cloud.

Store test cases in the cloud, trigger runs, and analyze results with full runner logs, screenshots, and trace files.

Cloud tools require SHIPLIGHT_API_TOKEN

Cloud sync tools (save_test_case, get_test_case, save_template, get_template, save_function, get_function) are automatically enabled when SHIPLIGHT_API_TOKEN is set. No special flags needed — just add the token to your project's .env file or MCP config env.

Get your token from app.shiplight.ai/settings/api-tokens.

SHIPLIGHT_API_TOKEN is resolved in this order: MCP config env, then inherited host environment variables, then .env in the current working directory.

Capabilities

The AI agent can:

• Create and update test cases from YAML E2E test flows
• Trigger cloud test runs across environments
• Get detailed results: step-by-step status, screenshots, runner logs
• Manage test infrastructure: environments, test accounts, folders

Example conversation:

You: Create a test case from my login-test.yaml and run it on staging

Claude: [reads YAML, creates test case, triggers run, polls for results]
Test case #502 created. Cloud run completed in 1m 23s — all 8 steps passed.

Test Case Management

Tool	Description
save_test_case	Save a YAML flow — auto-creates or updates based on embedded test_case_id
get_test_case	Get test case details (supports YAML output with test_case_id embedded)

Test Execution

Tool	Description
run_test_case	Trigger a cloud test run
list_test_runs	List test runs with filtering
get_test_run_details	Get run status and test case results

Test Results

Tool	Description
get_test_case_result	Get detailed result with runner logs (stdout/stderr)
get_test_case_result_steps	Get step-by-step execution details
get_step_artifacts	Download screenshots and artifacts for a step

Infrastructure

Tool	Description
list_environments	List testing environments
list_test_accounts	List test accounts for an environment
get_test_account	Get test account details
create_test_account	Create a test account with login config
list_folders	List test case folders
create_folder	Create a folder for organizing test cases
get_folder	Get folder details with full path

Testing Your Local Dev Server from the Cloud

Cloud test runners cannot access localhost directly. To test your local development server from the cloud, use a tunneling service to expose it via a public HTTPS URL:

ngrok

ngrok http 3000
# Gives you a URL like https://abc123.ngrok-free.app

pinggy

ssh -p 443 -R0:localhost:3000 a.pinggy.io
# Gives you a URL like https://abc123.a.pinggy.link

Then use the tunnel URL as your environment URL in Shiplight Cloud:

Run my login test case on https://abc123.ngrok-free.app

TIP

Tunneling services work well for quick ad-hoc testing. For regular use, consider deploying to a staging environment instead.

Running Tests Locally

Test flows can also be exported as standalone Playwright tests for running on your own machine or in CI — no cloud runner needed. See Local Testing.