Test Editor Overview

View as Markdown

LEGACY — Shiplight Cloud v1

This page documents Shiplight Cloud v1 (app.shiplight.ai), which is no longer the current product. For the current cloud platform, see Shiplight Cloud.

The Shiplight AI Test Editor is a powerful, AI-enhanced interface for creating, editing, and debugging automated tests. It provides both natural language and code-based test authoring capabilities with real-time execution and debugging features.

Table of Contents

1. Accessing the Test Editor
2. Editor Interface Overview
   • Header Section
   • Main Editor Area
3. Test Structure
   • Test Flow
   • Step Types
   • Teardown Section
4. Key Features
   • Natural Language & Code Modes
   • AI Mode vs Fast Mode
   • Recording
   • Test Information Panels
5. Editing Workflow
6. Status Management
   • Test Status Types
   • Status Transitions
7. Best Practices
   • Test Design
   • Performance Tips
   • Maintenance
8. Keyboard Shortcuts
9. Next Steps

Accessing the Test Editor

Navigate to the Test Editor through multiple entry points:

1. Automatic Redirect - Opens automatically after creating a new test
2. Tests Page - Click on any test from the main Tests page
3. Test Suites - Click on a test within a suite
4. Quick Access - Use the arrow (→) button wherever tests are listed
5. Direct URL - Navigate to /tests/{testId}

Editor Interface Overview

Header Section

The test editor header provides essential test information and controls:

• Test Name - Click to edit the test name inline
• Status Badge - Shows current test status:
  • Draft - In development, excluded from scheduled runs
  • Active - Enabled for automated execution
  • Generating - AI is creating test steps
• Environment Selector - Choose which environment to run the test in
• Actions Menu - Additional options (clone, delete, export)

Main Editor Area

The editor is divided into two resizable panels:

Left Panel - Test Flow Editor

• Visual representation of your test steps
• Drag-and-drop step reordering
• Inline editing capabilities
• AI mode toggle for natural language input
• Nested structure support (steps, conditions, loops)

Right Panel - Live View & Information

• Live Tab - Real-time browser view during debugging
• Preview Tab - Screenshots from test execution
• Console Tab - Execution logs and debug output
• Context Tab - Current test variables and state

Test Structure

Test Flow

A test flow contains:

• Main Statements - Your primary test logic
• Teardown Statements - Optional cleanup steps
• Metadata - Goal, URL, and completion status

Step Types

Each step in your test can be one of these types:

Action Step - A single operation (click, type, assert, etc.)

• Contains the actual action to perform
• The leaf node of your test structure

Group - A container for multiple steps

• Groups related steps together
• Can be collapsed/expanded for organization
• Contains nested statements

If/Else Step - Conditional branching

• Evaluates a condition
• Executes different paths based on result

Loop Step - Repeating operations

• While loop that continues based on condition
• Contains body statements to repeat

Teardown Section

Optional cleanup steps that run after the main test:

• Always executes (even on test failure)
• Useful for resetting state
• Contains the same step types as main flow
• Separate from main statements

Key Features

Natural Language & Code Modes

Natural Language Mode (AI-Powered)

• Write test steps in plain English
• AI automatically converts to executable actions
• Ideal for non-technical users
• Context-aware suggestions

Code Mode (JavaScript)

• Direct Playwright code editing
• Full access to browser APIs
• Custom logic implementation
• Advanced automation scenarios

AI Mode vs Fast Mode

Fast Mode (Non-AI)

• Uses cached actions for optimal performance
• Executes pre-generated Playwright commands
• Best for stable DOM elements and consistent UIs
• Faster execution speed
• Relies on fixed element selectors/locators
• Less flexible but more performant

AI Mode (Dynamic Mode)

• Evaluates descriptions in real-time against current browser state
• Dynamically determines the best action and element to target
• Handles dynamic element IDs and changing locators
• More flexible and adaptive to UI changes
• Slower execution due to real-time analysis
• Ideal for modern web apps with dynamic content

Recording

Interactive Test Creation

• Record browser interactions in real-time
• Automatically captures clicks, typing, navigation, and form interactions
• Converts user actions into test steps automatically
• Available during debugging sessions via Live View toolbar

Benefits:

• Rapid test creation by performing actions naturally
• No need to write step descriptions manually
• Captures complex user workflows accurately
• Ideal for exploratory testing and reproducing bugs

Learn more about Recording

Test Information Panels

Goal Section

• Document the test's purpose
• Editable at any time
• Used by AI for context
• Helps team understand test intent

Labels

• Organize tests with custom tags
• Filter and search by labels
• Apply multiple labels per test
• Color-coded for quick identification

Latest Run Results

• Last execution status
• Execution time and date
• Link to full run details
• Video recording (if available)

Editing Workflow

1. Adding Steps

Click "Add a new step" button to insert steps:

• Choose from action type dropdown
• Enter description or code
• Configure step-specific options
• AI auto-suggests next steps

Insert Between Steps using the + button:

• Maintains test flow
• Automatically adjusts numbering
• Preserves nested structure

2. Modifying Steps

Click the ⋯ menu on any step to access four groups of options:

Actions - Convert to different action types:

• Action, Assertion, Code, Function, Upload File, Group

Control Flows - Convert to control structures:

• If/Else, Loop

Execution - Debug controls:

• Run until here

Edit Operations:

• Duplicate step
• Delete step

3. Organizing Steps

Drag and Drop to reorder:

• Visual indicators show valid drop zones
• Maintains parent-child relationships
• Updates references automatically

Group Related Steps:

• Create step containers
• Add descriptive group names
• Collapse/expand for clarity
• Nest multiple levels deep

Status Management

Test Status Types

Draft

• Development and testing phase
• Not included in scheduled runs
• Can still be run manually
• No notifications on failure

Active

• Production-ready test
• Included in test suites and schedules
• Triggers notifications on failure
• Monitored for reliability

Disabled

• Temporarily excluded from execution
• Preserves test configuration
• Quick re-enable when needed
• Useful for maintenance windows

Status Transitions

Toggle status using the dropdown in the header:

1. Create test → Starts as Draft
2. Complete development → Change to Active
3. Maintenance needed → Set to Disabled
4. Updates complete → Return to Active

Best Practices

Test Design

✅ DO:

• Keep tests focused on single features
• Use descriptive step descriptions
• Add assertions to verify outcomes
• Implement proper wait strategies
• Use teardown for cleanup

❌ DON'T:

• Create overly complex test flows
• Hard-code environment-specific values
• Skip error handling
• Ignore flaky test warnings
• Duplicate test logic

Performance Tips

• Minimize Waits - Use smart waits instead of fixed delays
• Batch Operations - Group related actions together
• Reuse Functions - Extract common patterns
• Optimize Selectors - Use stable, efficient selectors
• Parallel Execution - Design tests for independence

Maintenance

• Regular Reviews - Update tests with UI changes
• Monitor Flakiness - Address intermittent failures
• Version Control - Save checkpoints before major changes
• Documentation - Keep goals and descriptions current
• Refactoring - Simplify complex flows periodically

Keyboard Shortcuts

When a statement is selected, you can use these shortcuts:

Shortcut	Action
Ctrl + Alt + A	Convert to Action
Ctrl + Alt + T	Convert to Assertion
Ctrl + Alt + C	Convert to Code
Ctrl + Alt + F	Convert to Function
Ctrl + Alt + U	Convert to Upload File
Ctrl + Alt + E	Convert to Extract
Ctrl + Alt + L	Convert to Login
Ctrl + Alt + W	Convert to Wait Until
Ctrl + Alt + G	Convert to Group
Ctrl + Alt + R	Convert to Template
Ctrl/Cmd + D	Duplicate statement

Next Steps

• Configure Test Environments - Set up multiple configurations for different environments
• Learn about Actions & Steps - All available action types and how to use them
• Explore Recording - Interactive test creation
• Explore AI Features - AI-powered test creation and maintenance
• Master Debugging Tools - Interactive debugging and troubleshooting
• Discover Advanced Features - Variables, functions, and more