# GitHub Actions Integration

<div class="view-markdown-wrapper">
<ViewMarkdown />
</div>

This guide helps you integrate Shiplight AI's automated testing into your GitHub CI/CD pipeline.

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Step 1: Get Your Shiplight API Token](#step-1-get-your-shiplight-api-token)
3. [Step 2: Add API Token to GitHub Secrets](#step-2-add-api-token-to-github-secrets)
4. [Step 3: Find Your Shiplight IDs](#step-3-find-your-shiplight-ids)
   - [Test Suite IDs](#test-suite-ids)
   - [Environment ID](#environment-id)
5. [Step 4: Create GitHub Workflow](#step-4-create-github-workflow)
   - [Basic Configuration (Pull Requests)](#basic-configuration-pull-requests)
   - [Advanced Configuration (With Deployment)](#advanced-configuration-with-deployment)
   - [Vercel Deployment Integration](#vercel-deployment-integration)
6. [Configuration Options](#configuration-options)
   - [Multiple Test Suites](#multiple-test-suites)
   - [Test Context](#test-context)
   - [Upgrading from v1 to v2.0.0](#upgrading-from-v1-to-v2)
7. [Action Outputs](#action-outputs)
   - [Using Outputs](#using-outputs)
8. [Test Account Configuration](#test-account-configuration)
9. [Troubleshooting](#troubleshooting)
   - [Workflow Not Running](#workflow-not-running)
   - [Authentication Failed](#authentication-failed)
   - [No PR Comments](#no-pr-comments)
   - [Tests Timing Out](#tests-timing-out)
   - [Multiple Suites Not Running](#multiple-suites-not-running)
10. [Best Practices](#best-practices)
11. [Support](#support)

## Prerequisites

Before starting, you'll need:

- Admin access to your GitHub repository (to add secrets and workflows)
- A Shiplight AI account with test suites configured
- Your Shiplight API token

## Step 1: Get Your Shiplight API Token

1. Log into [Shiplight AI](https://app.shiplight.ai)
2. Navigate to **Settings → API Tokens**
3. Create a new API token or use an existing one
4. Copy the token and keep it secure

> ⚠️ **Important**: Never commit your API token directly to your repository.

## Step 2: Add API Token to GitHub Secrets

1. Go to your GitHub repository
2. Click **Settings** → **Secrets and variables** → **Actions**
3. Click **New repository secret**
4. Add your secret:
   - **Name**: `SHIPLIGHT_API_TOKEN`
   - **Secret**: Paste your API token
5. Click **Add secret**

## Step 3: Find Your Shiplight IDs

You'll need these values from your Shiplight account:

### Test Suite IDs

1. Navigate to your test suites in Shiplight
2. Note the ID(s) of the suite(s) you want to run
3. You can run multiple suites by using comma-separated IDs

### Environment ID

1. Go to **Settings → Environments**
2. Find your target environment
3. Copy the Environment ID

## Step 4: Create GitHub Workflow

Create a new file `.github/workflows/shiplight-test.yml` in your repository:

### Basic Configuration (Pull Requests)

```yaml
name: Shiplight AI Tests

on:
  pull_request:
    branches:
      - main
      - develop

# Required for commenting on pull requests
permissions: write-all

jobs:
  test:
    name: Run Shiplight Tests
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Run Shiplight Tests
        uses: ShiplightAI/github-action@v1
        with:
          api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
          test-suite-id: 123 # Single suite
          # test-suite-id: 123,456,789  # Multiple suites
          environment-id: 1
          # Optional: Override environment URL
          # environment-url: https://staging.example.com
```

### Advanced Configuration (With Deployment)

```yaml
name: Deploy and Test

on:
  pull_request:
    branches:
      - main

permissions: write-all

jobs:
  deploy-and-test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      # Your deployment step
      - name: Deploy to Preview
        id: deploy
        run: |
          # Your deployment script
          echo "preview-url=https://preview-$\{{ github.event.number }}.example.com" >> $GITHUB_OUTPUT

      - name: Run Shiplight Tests
        uses: ShiplightAI/github-action@v2.0.0
        with:
          api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
          test-suite-id: 123,456 # Multiple suites run in parallel
          environment-id: 1
          preflight-test-case-id: 123 # Optional, use if you need to run a preflight test case before the main tests
          environment-url: ${{ steps.deploy.outputs.preview-url }}
          timeout-seconds: 1800 # 30 minutes
          commit-sha: $\{{ github.event.pull_request.head.sha }}
```

### Vercel Deployment Integration

For Vercel deployments, use repository dispatch:

```yaml
name: Shiplight Tests on Vercel Deploy

on:
  repository_dispatch:
    types:
      - "vercel.deployment.success"

permissions: write-all

jobs:
  test:
    # Only run for preview deployments
    if: github.event.client_payload.environment == 'preview'
    runs-on: ubuntu-latest

    steps:
      - name: Set commit status - pending
        uses: actions/github-script@v7
        with:
          script: |
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: context.payload.client_payload.git.sha,
              state: 'pending',
              context: 'Shiplight AI Tests',
              description: 'Running tests...',
              target_url: '$\{{ github.server_url }}/$\{{ github.repository }}/actions/runs/$\{{ github.run_id }}'
            });

      - name: Run Shiplight Tests
        uses: ShiplightAI/github-action@v2.0.0
        with:
          api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
          test-suite-id: 1,2,3 # Multiple suites
          environment-id: 1
          preflight-test-case-id: 123 # Optional, use if you need to run a preflight test case before the main tests
          environment-url: $\{{ github.event.client_payload.url }}
          commit-sha: $\{{ github.event.client_payload.git.sha }}
          timeout-seconds: 1800

      - name: Set commit status - complete
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const jobStatus = '${{ job.status }}';
            const state = jobStatus === 'success' ? 'success' : 'failure';
            const description = jobStatus === 'success' 
              ? 'Tests passed' 
              : 'Tests failed';

            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: context.payload.client_payload.git.sha,
              state: state,
              context: 'Shiplight AI Tests',
              description: description,
              target_url: '$\{{ github.server_url }}/$\{{ github.repository }}/actions/runs/$\{{ github.run_id }}'
            });
```

## Configuration Options

| Option                   | Required | Default        | Description                                                                      |
| ------------------------ | -------- | -------------- | -------------------------------------------------------------------------------- |
| `api-token`              | ✅ Yes   | -              | Your Shiplight API token                                                         |
| `test-suite-id`          | ✅ Yes   | -              | Single ID or comma-separated list (e.g., "1" or "1,2,3")                         |
| `environment-id`         | ✅ Yes   | -              | Shiplight environment ID                                                         |
| `preflight-test-case-id` | No       | -              | Shiplight preflight test case ID, run before the main tests                      |
| `environment-url`        | No       | -              | Override the environment's configured URL                                        |
| `github-comment`         | No       | `true`         | Comment on pull requests with results                                            |
| `github-token`           | No       | `GITHUB_TOKEN` | Token for PR comments (auto-provided)                                            |
| `async`                  | No       | `false`        | Launch tests without waiting (disables comments)                                 |
| `commit-sha`             | No       | -              | Specific commit SHA for finding the PR                                           |
| `timeout-seconds`        | No       | `86400`        | Maximum wait time (default: 24 hours)                                            |
| `test-context`           | No       | -              | Multi-line key-value context passed to the test run (e.g. env, branch, build-id) |

### Multiple Test Suites

The action supports running multiple test suites in parallel for efficiency:

```yaml
test-suite-id: 1        # Single suite
test-suite-id: 1,2,3    # Multiple suites (run in parallel)
```

### Test Context

**Available in v2.0.0+.** Use `test-context` to pass key-value context (e.g. environment, branch, build ID, API keys) into your test run. The value is multi-line; each line is typically `key=value`.

```yaml
- name: Run Shiplight Tests
  uses: ShiplightAI/github-action@v2.0.0
  with:
    api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
    test-suite-id: 123
    environment-id: 1
    test-context: |
      env=${{ vars.DEPLOY_ENV }}
      branch=${{ github.ref_name }}
      build-id=${{ github.run_id }}
      api-key=${{ secrets.STAGING_API_KEY }}
```

### Preflight Test Case

The action supports running a preflight test case before the main tests. The preflight runs first; only when it passes will the main test suite(s) run.

```yaml
preflight-test-case-id: 123 # Optional, run before the main tests
```

**When to use a preflight test case**

| Scenario                           | Description                                                                                                                                                                               |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Smoke / health check**           | Run a single quick test (e.g. login or homepage load) to confirm the environment is up and reachable before running the full suite. Saves time and credits when the deployment is broken. |
| **Gate before heavy tests**        | Use one lightweight test as a gate: if it fails, skip the rest. Reduces noise from cascading failures and speeds up feedback when the app is clearly broken.                              |
| **Auth / session setup**           | Use a preflight that performs login or session setup; main tests can assume an authenticated state and focus on feature flows.                                                            |
| **Preview / ephemeral URLs**       | After deploying to a preview URL, run a preflight that hits that URL to verify the deployment succeeded before running the full suite against it.                                         |
| **Flaky or order-dependent tests** | Isolate a critical path (e.g. “can we complete one full flow?”) in the preflight so the main suite runs only when that path is stable.                                                    |

### Upgrading from v1 to v2.0.0

If you are currently using the action at `@v1` and want to use **preflight test cases** (`preflight-test-case-id`) or **test context** (`test-context`), upgrade to `v2.0.0` as follows.

**What changes**

| Item              | v1                             | v2.0.0                                         |
| ----------------- | ------------------------------ | ---------------------------------------------- |
| Action reference  | `ShiplightAI/github-action@v1` | `ShiplightAI/github-action@v2.0.0`             |
| Preflight support | Not available                  | Optional `preflight-test-case-id`              |
| Test context      | Not available                  | Optional `test-context` (multi-line key-value) |

**Steps to upgrade**

1. **Update the action version** in your workflow file (e.g. `.github/workflows/shiplight-test.yml`):

   ```yaml
   # Before (v1)
   - name: Run Shiplight Tests
     uses: ShiplightAI/github-action@v1

   # After (v2.0.0)
   - name: Run Shiplight Tests
     uses: ShiplightAI/github-action@v2.0.0
   ```

2. **(Optional) Add preflight** — To run a single test case before your main suites, add `preflight-test-case-id` with the test case ID from Shiplight:

   ```yaml
   - name: Run Shiplight Tests
     uses: ShiplightAI/github-action@v2.0.0
     with:
       api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
       test-suite-id: 123,456
       environment-id: 1
       preflight-test-case-id: 789 # optional: run before main tests
   ```

3. **(Optional) Add test context** — To pass env, branch, build-id, or other key-value context into your test run, add `test-context` (see [Test Context](#test-context)).

4. **No other changes required** — Existing inputs (`api-token`, `test-suite-id`, `environment-id`, `environment-url`, `timeout-seconds`, etc.) work the same in v2.0.0. You can upgrade first and add `preflight-test-case-id` or `test-context` later when needed.

**Summary**

- Replace `@v1` with `@v2.0.0` in the `uses` line.
- Optionally set `preflight-test-case-id` to run a preflight before the main test suites.
- Optionally set `test-context` to pass key-value context (env, branch, build-id, etc.) into the test run.

## Action Outputs

The action provides outputs you can use in subsequent steps:

| Output    | Description                                                 |
| --------- | ----------------------------------------------------------- |
| `success` | Whether all test runs were successful (`true`/`false`)      |
| `results` | JSON string containing detailed results for each test suite |

### Using Outputs

```yaml
- name: Run Shiplight Tests
  id: shiplight
  uses: ShiplightAI/github-action@v1
  with:
    api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }}
    test-suite-id: 1,2
    environment-id: 1

- name: Check Results
  if: steps.shiplight.outputs.success == 'false'
  run: |
    echo "Tests failed!"
    echo "Results: ${{ steps.shiplight.outputs.results }}"
    exit 1
```

## Test Account Configuration

If your tests require authentication, configure test accounts in Shiplight:

1. Navigate to **Settings → Test Accounts**
2. Click **Add Test Account**
3. Select the environment (scope)
4. Enter credentials
5. Save the account

The test runner will automatically use the appropriate test account based on the environment ID.

## Troubleshooting

### Workflow Not Running

- Verify the workflow file is in `.github/workflows/`
- Check that the file extension is `.yml` or `.yaml`
- Ensure the trigger event matches your needs

### Authentication Failed

- Verify the API token is correct
- Check that the secret name is exactly `SHIPLIGHT_API_TOKEN`
- Ensure the token has necessary permissions

### No PR Comments

- Add `permissions: write-all` to your workflow
- Verify `github-comment` is not set to `false`
- Check that the action is running on a pull request event

### Tests Timing Out

- Increase `timeout-seconds` value
- Check if your test suites are too large
- Consider splitting into smaller suites

### Multiple Suites Not Running

- Ensure you're using the latest action version (`@v1` or later)
- Verify suite IDs are comma-separated without spaces

## Best Practices

1. **Use Multiple Suites**: Run related tests in parallel for faster feedback
2. **Set Appropriate Timeouts**: Balance between allowing enough time and failing fast
3. **Environment URLs**: Use dynamic URLs for preview deployments
4. **Commit SHA**: Specify for accurate PR matching in complex workflows
5. **Monitor Usage**: Check your Shiplight dashboard for test execution metrics

## Support

- Check test results: [Shiplight Dashboard](https://app.shiplight.ai)
- Get help: Contact your Shiplight support team