CI Runners

View as Markdown

Shiplight runners are an alternative to GitHub-hosted runners like ubuntu-latest. They are ephemeral VMs that register themselves with GitHub as standard self-hosted runners, run exactly one job, and are then destroyed.

They are not a separate CI system. Your workflow files stay as they are — the only line that changes is runs-on. This page covers the cloud side: enabling runners, choosing a size, managing pools, and debugging dispatches. For writing the workflow itself, see GitHub Actions.

Why use them

	GitHub-hosted ubuntu-latest	Shiplight runner
Browsers	You run npx playwright install	Chromium, Firefox, and WebKit prebaked
Shiplight credentials	You store SHIPLIGHT_API_TOKEN as a secret	Injected per run, revoked when the run ends
LLM credentials	You supply your own provider key	LLM proxy credentials injected by default — your own key still works
Result upload	You set SHIPLIGHT_REPORT_TO_CLOUD=1	Enabled by the platform
Machine size	Fixed	4, 8, 16, or 32 vCPU

Setup

1. Install the GitHub App

An org owner installs the Shiplight GitHub App from Settings → Integrations → Connect GitHub at app.shiplight.ai/settings/integrations. The consent screen requests:

Permission	Level	Why
Actions	Read-only	Receive workflow_job events
Self-hosted runners (organization)	Read and write	Mint runner registration tokens
Metadata	Read-only	Basic repository data
Contents	Read-only	Check out your code
Pull requests	Read-only	Associate runs with PRs
Checks	Read and write	Report status back

The App also subscribes to the Workflow job webhook event.

The repository must be owned by a GitHub organization

Runner registration tokens are minted at the organization level. Repositories under a personal GitHub account cannot use Shiplight runners. The job sits queued in GitHub indefinitely and nothing is recorded under Dispatches, which makes this the most common — and least obvious — setup trap.

2. Connect the repository

The repository must appear as an active connected repo under Settings → Integrations. If you install the App but exclude a repository from its repository selection, that repo's jobs are never run — the dispatch is closed as Cancelled by user, even though nobody cancelled it.

3. Enable runners for the org

An owner turns on the Shiplight runners toggle on the Pools tab at app.shiplight.ai/runners.

It is off by default, and only owners can change it. While it is off you are not charged: the job waits in GitHub's "Waiting for runner" state, and the dispatch is recorded under Organization not enabled.

Turning it off later does not kill running jobs; they finish, and only new queued events are ignored.

4. Point your workflow at a runner

jobs:
  e2e:
    runs-on: shiplight-medium

Runner sizes

runs-on label	vCPU	Memory
shiplight-small	4	16 GB
shiplight-medium	8	32 GB
shiplight-large	16	64 GB
shiplight-xlarge	32	128 GB

Exactly one shiplight-* size label must be present on runs-on.

Managing pools

The Pools tab at app.shiplight.ai/runners lists one row per size, each with:

• Enabled — whether that size can be provisioned at all
• Min — runners kept provisioned (default 0)
• Max — ceiling for that pool (default 5)

Only owners can edit pools, and changes apply on the next scaler tick. A pool that is disabled or has Max set to 0 provisions nothing, and matching jobs wait in GitHub indefinitely.

Limits

• Total concurrent runs are capped, at 20 by default. Contact the Shiplight team to raise it if you need more.
• Jobs are capped at 60 minutes by default, and killed with Time cap exceeded. A workflow declaring timeout-minutes: 90 is clamped to the ceiling — the higher value is not honored. Contact the Shiplight team if you need a longer ceiling.

Debugging a run

The Dispatches tab at app.shiplight.ai/runners is the first place to look when a job misbehaves — check it before escalating.

Filter by status (Queued / In Progress / Completed), repository, or date range. Each row shows status, repository, branch, start time, duration, and a link straight to the job on GitHub. Any row that isn't still queued expands to give you:

• The failure reason, in plain language
• An execution timeline of the GitHub Actions steps with per-step duration and outcome
• The linked Shiplight Run #N, with pass/fail/flaky/skipped counts and a link to the run details

Failure reasons

Shown as	What it means
Process exited with non-zero code	Your tests or a workflow step failed
Time cap exceeded	Hit the time ceiling
Multiple size labels detected	More than one shiplight-* label on runs-on
Unsupported size label	The label is not a known size
Organization not enabled	The Shiplight runners toggle is off
Cancelled by user	The job was cancelled — or the repository is not connected, which closes the dispatch the same way
GitHub assigned job elsewhere	GitHub gave the job to a different runner
Runner provisioning failed	A platform-side failure — contact support

Job stays queued

Some conditions produce no dispatch record at all and simply leave the job waiting in GitHub. Work through these in order:

1. The pool for that size is disabled, or its Max is 0.
2. The org-wide concurrency cap is saturated.
3. The repository is under a personal account rather than an organization.
4. Your account balance is insufficient.

If instead you see a dispatch with a failure reason, use the table above — a disconnected repository and a disabled organization both record one.

Bringing your own runner

Shiplight does not ship an installable runner agent. If you want to run on your own hardware, use GitHub's own self-hosted runners and treat them like any non-Shiplight runner: supply SHIPLIGHT_API_TOKEN yourself and set SHIPLIGHT_REPORT_TO_CLOUD=1 to upload results. GitHub Actions documents that path.

Related

• GitHub Actions — authoring the workflow file
• CI/CD overview — other CI providers
• API Tokens — tokens and scopes, including the LLM proxy
• Test Results — what happens to results after upload