GitHub Actions Integration

View as Markdown

This guide helps you integrate Shiplight AI's automated testing into your GitHub CI/CD pipeline.

Table of Contents

1. Prerequisites
2. Step 1: Get Your Shiplight API Token
3. Step 2: Add API Token to GitHub Secrets
4. Step 3: Find Your Shiplight IDs
   • Test Suite IDs
   • Environment ID
5. Step 4: Create GitHub Workflow
   • Basic Configuration (Pull Requests)
   • Advanced Configuration (With Deployment)
   • Vercel Deployment Integration
6. Configuration Options
   • Multiple Test Suites
7. Action Outputs
   • Using Outputs
8. Test Account Configuration
9. Troubleshooting
   • Workflow Not Running
   • Authentication Failed
   • No PR Comments
   • Tests Timing Out
   • Multiple Suites Not Running
10. Best Practices
11. Support

Prerequisites

Before starting, you'll need:

• Admin access to your GitHub repository (to add secrets and workflows)
• A Shiplight AI account with test suites configured
• Your Shiplight API token

Step 1: Get Your Shiplight API Token

1. Log into Shiplight AI
2. Navigate to Settings → API Tokens
3. Create a new API token or use an existing one
4. Copy the token and keep it secure

⚠️ Important: Never commit your API token directly to your repository.

Step 2: Add API Token to GitHub Secrets

1. Go to your GitHub repository
2. Click Settings → Secrets and variables → Actions
3. Click New repository secret
4. Add your secret:
   • Name: SHIPLIGHT_API_TOKEN
   • Secret: Paste your API token
5. Click Add secret

Step 3: Find Your Shiplight IDs

You'll need these values from your Shiplight account:

Test Suite IDs

1. Navigate to your test suites in Shiplight
2. Note the ID(s) of the suite(s) you want to run
3. You can run multiple suites by using comma-separated IDs

Environment ID

1. Go to Settings → Environments
2. Find your target environment
3. Copy the Environment ID

Step 4: Create GitHub Workflow

Create a new file .github/workflows/shiplight-test.yml in your repository:

Basic Configuration (Pull Requests)

name: Shiplight AI Tests

on:
  pull_request:
    branches:
      - main
      - develop

# Required for commenting on pull requests
permissions: write-all

jobs:
  test:
    name: Run Shiplight Tests
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Run Shiplight Tests
        uses: ShiplightAI/github-action@v1
        with:
          api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
          test-suite-id: 123 # Single suite
          # test-suite-id: 123,456,789  # Multiple suites
          environment-id: 1
          # Optional: Override environment URL
          # environment-url: https://staging.example.com

Advanced Configuration (With Deployment)

name: Deploy and Test

on:
  pull_request:
    branches:
      - main

permissions: write-all

jobs:
  deploy-and-test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      # Your deployment step
      - name: Deploy to Preview
        id: deploy
        run: |
          # Your deployment script
          echo "preview-url=https://preview-$\{{ github.event.number }}.example.com" >> $GITHUB_OUTPUT

      - name: Run Shiplight Tests
        uses: ShiplightAI/github-action@v1
        with:
          api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
          test-suite-id: 123,456 # Multiple suites run in parallel
          environment-id: 1
          environment-url: ${{ steps.deploy.outputs.preview-url }}
          timeout-seconds: 1800 # 30 minutes
          commit-sha: $\{{ github.event.pull_request.head.sha }}

Vercel Deployment Integration

For Vercel deployments, use repository dispatch:

name: Shiplight Tests on Vercel Deploy

on:
  repository_dispatch:
    types:
      - "vercel.deployment.success"

permissions: write-all

jobs:
  test:
    # Only run for preview deployments
    if: github.event.client_payload.environment == 'preview'
    runs-on: ubuntu-latest

    steps:
      - name: Set commit status - pending
        uses: actions/github-script@v7
        with:
          script: |
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: context.payload.client_payload.git.sha,
              state: 'pending',
              context: 'Shiplight AI Tests',
              description: 'Running tests...',
              target_url: '$\{{ github.server_url }}/$\{{ github.repository }}/actions/runs/$\{{ github.run_id }}'
            });

      - name: Run Shiplight Tests
        uses: ShiplightAI/github-action@v1
        with:
          api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
          test-suite-id: 1,2,3 # Multiple suites
          environment-id: 1
          environment-url: $\{{ github.event.client_payload.url }}
          commit-sha: $\{{ github.event.client_payload.git.sha }}
          timeout-seconds: 1800

      - name: Set commit status - complete
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const jobStatus = '${{ job.status }}';
            const state = jobStatus === 'success' ? 'success' : 'failure';
            const description = jobStatus === 'success' 
              ? 'Tests passed' 
              : 'Tests failed';

            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: context.payload.client_payload.git.sha,
              state: state,
              context: 'Shiplight AI Tests',
              description: description,
              target_url: '$\{{ github.server_url }}/$\{{ github.repository }}/actions/runs/$\{{ github.run_id }}'
            });

Configuration Options

Option	Required	Default	Description
api-token	✅ Yes	-	Your Shiplight API token
test-suite-id	✅ Yes	-	Single ID or comma-separated list (e.g., "1" or "1,2,3")
environment-id	✅ Yes	-	Shiplight environment ID
environment-url	No	-	Override the environment's configured URL
github-comment	No	true	Comment on pull requests with results
github-token	No	GITHUB_TOKEN	Token for PR comments (auto-provided)
async	No	false	Launch tests without waiting (disables comments)
commit-sha	No	-	Specific commit SHA for finding the PR
timeout-seconds	No	86400	Maximum wait time (default: 24 hours)

Multiple Test Suites

The action supports running multiple test suites in parallel for efficiency:

test-suite-id: 1        # Single suite
test-suite-id: 1,2,3    # Multiple suites (run in parallel)

Action Outputs

The action provides outputs you can use in subsequent steps:

Output	Description
success	Whether all test runs were successful (true/false)
results	JSON string containing detailed results for each test suite

Using Outputs

- name: Run Shiplight Tests
  id: shiplight
  uses: ShiplightAI/github-action@v1
  with:
    api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
    test-suite-id: 1,2
    environment-id: 1

- name: Check Results
  if: steps.shiplight.outputs.success == 'false'
  run: |
    echo "Tests failed!"
    echo "Results: ${{ steps.shiplight.outputs.results }}"
    exit 1

Test Account Configuration

If your tests require authentication, configure test accounts in Shiplight:

1. Navigate to Settings → Test Accounts
2. Click Add Test Account
3. Select the environment (scope)
4. Enter credentials
5. Save the account

The test runner will automatically use the appropriate test account based on the environment ID.

Troubleshooting

Workflow Not Running

• Verify the workflow file is in .github/workflows/
• Check that the file extension is .yml or .yaml
• Ensure the trigger event matches your needs

Authentication Failed

• Verify the API token is correct
• Check that the secret name is exactly SHIPLIGHT_API_TOKEN
• Ensure the token has necessary permissions

No PR Comments

• Add permissions: write-all to your workflow
• Verify github-comment is not set to false
• Check that the action is running on a pull request event

Tests Timing Out

• Increase timeout-seconds value
• Check if your test suites are too large
• Consider splitting into smaller suites

Multiple Suites Not Running

• Ensure you're using the latest action version (@v1 or later)
• Verify suite IDs are comma-separated without spaces

Best Practices

1. Use Multiple Suites: Run related tests in parallel for faster feedback
2. Set Appropriate Timeouts: Balance between allowing enough time and failing fast
3. Environment URLs: Use dynamic URLs for preview deployments
4. Commit SHA: Specify for accurate PR matching in complex workflows
5. Monitor Usage: Check your Shiplight dashboard for test execution metrics

Support

• Check test results: Shiplight Dashboard
• Get help: Contact your Shiplight support team