GitHub Actions Integration

View as Markdown

This guide helps you integrate Shiplight AI's automated testing into your GitHub CI/CD pipeline.

Table of Contents

1. Prerequisites
2. Step 1: Get Your Shiplight API Token
3. Step 2: Add API Token to GitHub Secrets
4. Step 3: Find Your Shiplight IDs
   • Test Suite IDs
   • Environment ID
5. Step 4: Create GitHub Workflow
   • Basic Configuration (Pull Requests)
   • Advanced Configuration (With Deployment)
   • Vercel Deployment Integration
6. Configuration Options
   • Multiple Test Suites
   • Test Context
   • Upgrading from v1 to v2.0.0
7. Action Outputs
   • Using Outputs
8. Test Account Configuration
9. Troubleshooting
   • Workflow Not Running
   • Authentication Failed
   • No PR Comments
   • Tests Timing Out
   • Multiple Suites Not Running
10. Best Practices
11. Support

Prerequisites

Before starting, you'll need:

• Admin access to your GitHub repository (to add secrets and workflows)
• A Shiplight AI account with test suites configured
• Your Shiplight API token

Step 1: Get Your Shiplight API Token

1. Log into Shiplight AI
2. Navigate to Settings → API Tokens
3. Create a new API token or use an existing one
4. Copy the token and keep it secure

⚠️ Important: Never commit your API token directly to your repository.

Step 2: Add API Token to GitHub Secrets

1. Go to your GitHub repository
2. Click Settings → Secrets and variables → Actions
3. Click New repository secret
4. Add your secret:
   • Name: SHIPLIGHT_API_TOKEN
   • Secret: Paste your API token
5. Click Add secret

Step 3: Find Your Shiplight IDs

You'll need these values from your Shiplight account:

Test Suite IDs

1. Navigate to your test suites in Shiplight
2. Note the ID(s) of the suite(s) you want to run
3. You can run multiple suites by using comma-separated IDs

Environment ID

1. Go to Settings → Environments
2. Find your target environment
3. Copy the Environment ID

Step 4: Create GitHub Workflow

Create a new file .github/workflows/shiplight-test.yml in your repository:

Basic Configuration (Pull Requests)

name: Shiplight AI Tests

on:
  pull_request:
    branches:
      - main
      - develop

# Required for commenting on pull requests
permissions: write-all

jobs:
  test:
    name: Run Shiplight Tests
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Run Shiplight Tests
        uses: ShiplightAI/github-action@v1
        with:
          api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
          test-suite-id: 123 # Single suite
          # test-suite-id: 123,456,789  # Multiple suites
          environment-id: 1
          # Optional: Override environment URL
          # environment-url: https://staging.example.com

Advanced Configuration (With Deployment)

name: Deploy and Test

on:
  pull_request:
    branches:
      - main

permissions: write-all

jobs:
  deploy-and-test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      # Your deployment step
      - name: Deploy to Preview
        id: deploy
        run: |
          # Your deployment script
          echo "preview-url=https://preview-$\{{ github.event.number }}.example.com" >> $GITHUB_OUTPUT

      - name: Run Shiplight Tests
        uses: ShiplightAI/github-action@v2.0.0
        with:
          api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
          test-suite-id: 123,456 # Multiple suites run in parallel
          environment-id: 1
          preflight-test-case-id: 123 # Optional, use if you need to run a preflight test case before the main tests
          environment-url: ${{ steps.deploy.outputs.preview-url }}
          timeout-seconds: 1800 # 30 minutes
          commit-sha: $\{{ github.event.pull_request.head.sha }}

Vercel Deployment Integration

For Vercel deployments, use repository dispatch:

name: Shiplight Tests on Vercel Deploy

on:
  repository_dispatch:
    types:
      - "vercel.deployment.success"

permissions: write-all

jobs:
  test:
    # Only run for preview deployments
    if: github.event.client_payload.environment == 'preview'
    runs-on: ubuntu-latest

    steps:
      - name: Set commit status - pending
        uses: actions/github-script@v7
        with:
          script: |
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: context.payload.client_payload.git.sha,
              state: 'pending',
              context: 'Shiplight AI Tests',
              description: 'Running tests...',
              target_url: '$\{{ github.server_url }}/$\{{ github.repository }}/actions/runs/$\{{ github.run_id }}'
            });

      - name: Run Shiplight Tests
        uses: ShiplightAI/github-action@v2.0.0
        with:
          api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
          test-suite-id: 1,2,3 # Multiple suites
          environment-id: 1
          preflight-test-case-id: 123 # Optional, use if you need to run a preflight test case before the main tests
          environment-url: $\{{ github.event.client_payload.url }}
          commit-sha: $\{{ github.event.client_payload.git.sha }}
          timeout-seconds: 1800

      - name: Set commit status - complete
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const jobStatus = '${{ job.status }}';
            const state = jobStatus === 'success' ? 'success' : 'failure';
            const description = jobStatus === 'success' 
              ? 'Tests passed' 
              : 'Tests failed';

            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: context.payload.client_payload.git.sha,
              state: state,
              context: 'Shiplight AI Tests',
              description: description,
              target_url: '$\{{ github.server_url }}/$\{{ github.repository }}/actions/runs/$\{{ github.run_id }}'
            });

Configuration Options

Option	Required	Default	Description
api-token	✅ Yes	-	Your Shiplight API token
test-suite-id	✅ Yes	-	Single ID or comma-separated list (e.g., "1" or "1,2,3")
environment-id	✅ Yes	-	Shiplight environment ID
preflight-test-case-id	No	-	Shiplight preflight test case ID, run before the main tests
environment-url	No	-	Override the environment's configured URL
github-comment	No	true	Comment on pull requests with results
github-token	No	GITHUB_TOKEN	Token for PR comments (auto-provided)
async	No	false	Launch tests without waiting (disables comments)
commit-sha	No	-	Specific commit SHA for finding the PR
timeout-seconds	No	86400	Maximum wait time (default: 24 hours)
test-context	No	-	Multi-line key-value context passed to the test run (e.g. env, branch, build-id)

Multiple Test Suites

The action supports running multiple test suites in parallel for efficiency:

test-suite-id: 1        # Single suite
test-suite-id: 1,2,3    # Multiple suites (run in parallel)

Test Context

Available in v2.0.0+. Use test-context to pass key-value context (e.g. environment, branch, build ID, API keys) into your test run. The value is multi-line; each line is typically key=value.

- name: Run Shiplight Tests
  uses: ShiplightAI/github-action@v2.0.0
  with:
    api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
    test-suite-id: 123
    environment-id: 1
    test-context: |
      env=${{ vars.DEPLOY_ENV }}
      branch=${{ github.ref_name }}
      build-id=${{ github.run_id }}
      api-key=${{ secrets.STAGING_API_KEY }}

Preflight Test Case

The action supports running a preflight test case before the main tests. The preflight runs first; only when it passes will the main test suite(s) run.

preflight-test-case-id: 123 # Optional, run before the main tests

When to use a preflight test case

Scenario	Description
Smoke / health check	Run a single quick test (e.g. login or homepage load) to confirm the environment is up and reachable before running the full suite. Saves time and credits when the deployment is broken.
Gate before heavy tests	Use one lightweight test as a gate: if it fails, skip the rest. Reduces noise from cascading failures and speeds up feedback when the app is clearly broken.
Auth / session setup	Use a preflight that performs login or session setup; main tests can assume an authenticated state and focus on feature flows.
Preview / ephemeral URLs	After deploying to a preview URL, run a preflight that hits that URL to verify the deployment succeeded before running the full suite against it.
Flaky or order-dependent tests	Isolate a critical path (e.g. “can we complete one full flow?”) in the preflight so the main suite runs only when that path is stable.

Upgrading from v1 to v2.0.0

If you are currently using the action at @v1 and want to use preflight test cases (preflight-test-case-id) or test context (test-context), upgrade to v2.0.0 as follows.

What changes

Item	v1	v2.0.0
Action reference	ShiplightAI/github-action@v1	ShiplightAI/github-action@v2.0.0
Preflight support	Not available	Optional preflight-test-case-id
Test context	Not available	Optional test-context (multi-line key-value)

Steps to upgrade

1. Update the action version in your workflow file (e.g. .github/workflows/shiplight-test.yml):

   # Before (v1)
   - name: Run Shiplight Tests
     uses: ShiplightAI/github-action@v1
   
   # After (v2.0.0)
   - name: Run Shiplight Tests
     uses: ShiplightAI/github-action@v2.0.0

2. (Optional) Add preflight — To run a single test case before your main suites, add preflight-test-case-id with the test case ID from Shiplight:

   - name: Run Shiplight Tests
     uses: ShiplightAI/github-action@v2.0.0
     with:
       api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
       test-suite-id: 123,456
       environment-id: 1
       preflight-test-case-id: 789 # optional: run before main tests

3. (Optional) Add test context — To pass env, branch, build-id, or other key-value context into your test run, add test-context (see Test Context).

4. No other changes required — Existing inputs (api-token, test-suite-id, environment-id, environment-url, timeout-seconds, etc.) work the same in v2.0.0. You can upgrade first and add preflight-test-case-id or test-context later when needed.

Summary

• Replace @v1 with @v2.0.0 in the uses line.
• Optionally set preflight-test-case-id to run a preflight before the main test suites.
• Optionally set test-context to pass key-value context (env, branch, build-id, etc.) into the test run.

Action Outputs

The action provides outputs you can use in subsequent steps:

Output	Description
success	Whether all test runs were successful (true/false)
results	JSON string containing detailed results for each test suite

Using Outputs

- name: Run Shiplight Tests
  id: shiplight
  uses: ShiplightAI/github-action@v1
  with:
    api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
    test-suite-id: 1,2
    environment-id: 1

- name: Check Results
  if: steps.shiplight.outputs.success == 'false'
  run: |
    echo "Tests failed!"
    echo "Results: ${{ steps.shiplight.outputs.results }}"
    exit 1

Test Account Configuration

If your tests require authentication, configure test accounts in Shiplight:

1. Navigate to Settings → Test Accounts
2. Click Add Test Account
3. Select the environment (scope)
4. Enter credentials
5. Save the account

The test runner will automatically use the appropriate test account based on the environment ID.

Troubleshooting

Workflow Not Running

• Verify the workflow file is in .github/workflows/
• Check that the file extension is .yml or .yaml
• Ensure the trigger event matches your needs

Authentication Failed

• Verify the API token is correct
• Check that the secret name is exactly SHIPLIGHT_API_TOKEN
• Ensure the token has necessary permissions

No PR Comments

• Add permissions: write-all to your workflow
• Verify github-comment is not set to false
• Check that the action is running on a pull request event

Tests Timing Out

• Increase timeout-seconds value
• Check if your test suites are too large
• Consider splitting into smaller suites

Multiple Suites Not Running

• Ensure you're using the latest action version (@v1 or later)
• Verify suite IDs are comma-separated without spaces

Best Practices

1. Use Multiple Suites: Run related tests in parallel for faster feedback
2. Set Appropriate Timeouts: Balance between allowing enough time and failing fast
3. Environment URLs: Use dynamic URLs for preview deployments
4. Commit SHA: Specify for accurate PR matching in complex workflows
5. Monitor Usage: Check your Shiplight dashboard for test execution metrics

Support

• Check test results: Shiplight Dashboard
• Get help: Contact your Shiplight support team