Webhook Integration Guide

View as Markdown

Overview

Shiplight Webhooks allow you to automatically receive test results when test runs complete. You can use this feature to:

• Integrate with internal monitoring systems
• Trigger automated workflows
• Send custom notifications
• Log test history to databases
• Integrate into CI/CD pipelines

---

Quick Start

1. Configure Webhook Endpoint (One-Time Setup)

First, you need to register your webhook endpoint in Shiplight's global settings:

1. Navigate to Settings → Notifications → Webhooks
2. Click Add Webhook Endpoint
3. Fill in the information:
   • Name: Give your webhook a descriptive name (e.g., "Production Dashboard", "Slack Notifier")
   • URL: Your server's webhook receiver URL (must be HTTPS, e.g., https://api.yourcompany.com/webhooks/shiplight)
   • Secret: Click "Generate Secret" to auto-generate (used for signature verification)
   • Enabled: Toggle on to activate this endpoint globally
4. Click Save
5. Important: Record the Secret somewhere safe (e.g., password manager) - you'll need it to verify webhook signatures on your server

Note: This is a one-time setup. Once configured, you can reuse this endpoint for multiple test plans and test suites.

2. Subscribe to Webhooks

After configuring endpoints, you can subscribe to receive notifications:

For Test Plans (Scheduled Runs)

1. Navigate to Test Plans → Select a Test Plan → Schedule Tab → Notifications
2. Click on the Webhook tab
3. Toggle on the webhook endpoint(s) you want to subscribe to
4. Configure Send When conditions:
   • All: Send for every test run (regardless of result)
   • Failed: Only send when there are failed test cases
   • Pass→Fail: Only send when tests regress (previously passed tests now fail)
   • Fail→Pass: Only send when tests are fixed (previously failed tests now pass)
5. Click Save

Trigger: Test plan webhooks trigger on Scheduled runs. The frontend automatically sets trigger = "Scheduled" when you configure the subscription.

For Test Suites (GitHub Action Runs)

1. Navigate to Test Suites → Select a Test Suite → Notifications Sidebar
2. Click on the Webhook tab in the notification modal
3. Toggle on the webhook endpoint(s) you want to subscribe to
4. Configure Send When conditions:
   • All: Send for every test run (regardless of result)
   • Failed: Only send when there are failed test cases
   • Passed: Only send when all tests pass
5. Click Save

Trigger: Test suite webhooks trigger on GitHub Action runs. The frontend automatically sets trigger = "GITHUB_ACTION" when you configure the subscription.

3. Implement Server-Side Receiver

Refer to the code examples below to implement your webhook endpoint that receives and processes test results.

---

Webhook Request Format

HTTP Headers

POST /your-webhook-endpoint HTTP/1.1
Host: your-server.com
Content-Type: application/json
User-Agent: Shiplight-Webhooks/1.0
X-Webhook-Signature: sha256=<HMAC-SHA256 signature>
X-Webhook-Timestamp: <Unix timestamp>

Request Body

Complete TestRunAnalysisResults JSON object:

{
  "testRun": {
    "id": 12345,
    "testPlanId": 678,
    "organizationId": "org-abc123",
    "status": "Finished",
    "result": "Failed",
    "trigger": "Scheduled",
    "totalTestCaseCount": 10,
    "passedTestCaseCount": 8,
    "failedTestCaseCount": 2,
    "skippedTestCaseCount": 0,
    "duration": 330,
    "startTime": "2025-12-07T10:24:30Z",
    "endTime": "2025-12-07T10:30:00Z",
    "target": "Production"
  },
  "testSuiteId": null,
  "passToFail": [
    {
      "id": 123,
      "testCaseId": 456,
      "title": "Login Flow",
      "result": "Failed",
      "url": "https://app.shiplight.ai/test-case-results/123"
    }
  ],
  "failToPass": [],
  "failedTestCases": [
    {
      "id": 123,
      "testCaseId": 456,
      "title": "Login Flow",
      "result": "Failed",
      "url": "https://app.shiplight.ai/test-case-results/123"
    },
    {
      "id": 124,
      "testCaseId": 457,
      "title": "Checkout Process",
      "result": "Failed",
      "url": "https://app.shiplight.ai/test-case-results/124"
    }
  ],
  "flakyTests": []
}

---

Field Descriptions

Root Level Fields

Field	Type	Description
testRun	object	Test run information (see testRun Object below)
testSuiteId	number | null	Test suite ID (only present for test suite runs, null for test plan runs)
passToFail	array	Test cases that changed from passing to failing (regressions)
failToPass	array	Test cases that changed from failing to passing (fixes)
failedTestCases	array	All failed test cases
flakyTests	array	Flaky test cases

testRun Object

Field	Type	Description
id	number	Test run ID
testPlanId	number | null	Test plan ID (only present for test plan runs, null for test suite runs)
organizationId	string	Organization ID
status	string	Run status (typically "Finished")
result	string	Test result: "Passed" or "Failed"
trigger	string	Trigger type: "Scheduled" (for test plans), "GITHUB_ACTION" (for test suites), "Manual", "API", etc.
totalTestCaseCount	number	Total test case count
passedTestCaseCount	number	Number of passed test cases
failedTestCaseCount	number	Number of failed test cases
skippedTestCaseCount	number	Number of skipped test cases
duration	number	Run duration (seconds)
startTime	string	Start time (ISO 8601)
endTime	string	End time (ISO 8601)
target	string	Target environment (optional)

Test Case Arrays

Each test case array (passToFail, failToPass, failedTestCases, flakyTests) contains objects with:

Field	Type	Description
id	number	Test case result ID
testCaseId	number	Test case ID
title	string	Test case title
result	string	Test result
url	string	Test result detail page URL (optional)

Distinguishing Test Plan vs Test Suite Webhooks

To determine whether a webhook is from a test plan or test suite run:

// Check if it's a test plan run
if (payload.testRun.testPlanId !== null && payload.testSuiteId === null) {
  console.log("This is a test plan run");
  console.log("Test Plan ID:", payload.testRun.testPlanId);
  console.log("Trigger:", payload.testRun.trigger); // Usually "Scheduled"
}

// Check if it's a test suite run
if (payload.testSuiteId !== null && payload.testRun.testPlanId === null) {
  console.log("This is a test suite run");
  console.log("Test Suite ID:", payload.testSuiteId);
  console.log("Trigger:", payload.testRun.trigger); // Usually "GITHUB_ACTION"
}

---

Signature Verification

Important: You must verify the request signature to ensure the request actually comes from Shiplight.

Signature Algorithm

1. Get X-Webhook-Timestamp header
2. Get request body (raw JSON string)
3. Concatenate: data = "{timestamp}.{body}"
4. Sign data with HMAC-SHA256 using your secret
5. Compare computed signature with X-Webhook-Signature header

Node.js Example

const crypto = require("crypto");
const express = require("express");
const app = express();

// Use express.text() instead of express.json() to get raw body
app.post("/webhooks/shiplight", express.text({ type: "application/json" }), (req, res) => {
  const signature = req.headers["x-webhook-signature"];
  const timestamp = req.headers["x-webhook-timestamp"];
  const secret = process.env.SHIPLIGHT_WEBHOOK_SECRET; // Read secret from environment variable

  // 1. Verify signature
  const data = `${timestamp}.${req.body}`;
  const hmac = crypto.createHmac("sha256", secret);
  hmac.update(data);
  const expectedSig = `sha256=${hmac.digest("hex")}`;

  if (signature !== expectedSig) {
    console.error("Invalid signature");
    return res.status(401).json({ error: "Invalid signature" });
  }

  // 2. Verify timestamp (prevent replay attacks)
  const now = Math.floor(Date.now() / 1000);
  const requestTime = parseInt(timestamp);
  if (Math.abs(now - requestTime) > 300) {
    // 5 minute tolerance
    console.error("Request too old");
    return res.status(401).json({ error: "Request too old" });
  }

  // 3. Parse and process data
  const payload = JSON.parse(req.body);
  const { testRun, failedTestCases, passToFail } = payload;

  console.log(`Test run ${testRun.id} finished with result: ${testRun.result}`);
  console.log(`Failed tests: ${failedTestCases.length}`);
  console.log(`Regressions: ${passToFail.length}`);

  // Your business logic
  // ...

  // 4. Return success response
  res.status(200).json({ received: true });
});

app.listen(3000, () => {
  console.log("Webhook server listening on port 3000");
});

Python (Flask) Example

import hmac
import hashlib
import time
import json
from flask import Flask, request, jsonify
import os

app = Flask(__name__)

@app.route('/webhooks/shiplight', methods=['POST'])
def handle_webhook():
    signature = request.headers.get('X-Webhook-Signature')
    timestamp = request.headers.get('X-Webhook-Timestamp')
    secret = os.environ['SHIPLIGHT_WEBHOOK_SECRET']

    # 1. Verify signature
    body = request.get_data(as_text=True)
    data = f"{timestamp}.{body}"
    expected_sig = 'sha256=' + hmac.new(
        secret.encode('utf-8'),
        data.encode('utf-8'),
        hashlib.sha256
    ).hexdigest()

    if not hmac.compare_digest(signature, expected_sig):
        return jsonify({'error': 'Invalid signature'}), 401

    # 2. Verify timestamp
    now = int(time.time())
    if abs(now - int(timestamp)) > 300:
        return jsonify({'error': 'Request too old'}), 401

    # 3. Parse and process data
    payload = json.loads(body)
    test_run = payload['testRun']
    failed_tests = payload['failedTestCases']
    regressions = payload['passToFail']

    print(f"Test run {test_run['id']} finished: {test_run['result']}")
    print(f"Failed tests: {len(failed_tests)}")
    print(f"Regressions: {len(regressions)}")

    # Your business logic
    # ...

    # 4. Return success response
    return jsonify({'received': True}), 200

if __name__ == '__main__':
    app.run(port=3000)

Go Example

package main

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "math"
    "net/http"
    "os"
    "strconv"
    "time"
)

type WebhookPayload struct {
    TestRun         TestRun         `json:"testRun"`
    PassToFail      []TestCase      `json:"passToFail"`
    FailToPass      []TestCase      `json:"failToPass"`
    FailedTestCases []TestCase      `json:"failedTestCases"`
    FlakyTests      []TestCase      `json:"flakyTests"`
}

type TestRun struct {
    ID                   int    `json:"id"`
    TestPlanID           int    `json:"testPlanId"`
    OrganizationID       string `json:"organizationId"`
    Status               string `json:"status"`
    Result               string `json:"result"`
    Trigger              string `json:"trigger"`
    TotalTestCaseCount   int    `json:"totalTestCaseCount"`
    PassedTestCaseCount  int    `json:"passedTestCaseCount"`
    FailedTestCaseCount  int    `json:"failedTestCaseCount"`
    SkippedTestCaseCount int    `json:"skippedTestCaseCount"`
    Duration             int    `json:"duration"`
    StartTime            string `json:"startTime"`
    EndTime              string `json:"endTime"`
    Target               string `json:"target"`
}

type TestCase struct {
    ID         int    `json:"id"`
    TestCaseID int    `json:"testCaseId"`
    Title      string `json:"title"`
    Result     string `json:"result"`
    URL        string `json:"url"`
}

func handleWebhook(w http.ResponseWriter, r *http.Request) {
    signature := r.Header.Get("X-Webhook-Signature")
    timestamp := r.Header.Get("X-Webhook-Timestamp")
    secret := os.Getenv("SHIPLIGHT_WEBHOOK_SECRET")

    // 1. Read body
    body, err := io.ReadAll(r.Body)
    if err != nil {
        http.Error(w, "Failed to read body", http.StatusBadRequest)
        return
    }

    // 2. Verify signature
    data := timestamp + "." + string(body)
    h := hmac.New(sha256.New, []byte(secret))
    h.Write([]byte(data))
    expectedSig := "sha256=" + hex.EncodeToString(h.Sum(nil))

    if !hmac.Equal([]byte(signature), []byte(expectedSig)) {
        http.Error(w, "Invalid signature", http.StatusUnauthorized)
        return
    }

    // 3. Verify timestamp
    requestTime, err := strconv.ParseInt(timestamp, 10, 64)
    if err != nil {
        http.Error(w, "Invalid timestamp", http.StatusBadRequest)
        return
    }
    now := time.Now().Unix()
    if math.Abs(float64(now-requestTime)) > 300 {
        http.Error(w, "Request too old", http.StatusUnauthorized)
        return
    }

    // 4. Parse payload
    var payload WebhookPayload
    if err := json.Unmarshal(body, &payload); err != nil {
        http.Error(w, "Invalid JSON", http.StatusBadRequest)
        return
    }

    // 5. Process data
    log.Printf("Test run %d finished: %s\n", payload.TestRun.ID, payload.TestRun.Result)
    log.Printf("Failed tests: %d\n", len(payload.FailedTestCases))
    log.Printf("Regressions: %d\n", len(payload.PassToFail))

    // Your business logic
    // ...

    // 6. Return success response
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(map[string]bool{"received": true})
}

func main() {
    http.HandleFunc("/webhooks/shiplight", handleWebhook)
    log.Println("Webhook server listening on :3000")
    log.Fatal(http.ListenAndServe(":3000", nil))
}

---

Testing Webhooks

Testing with webhook.site

1. Visit webhook.site
2. Copy the unique URL displayed on the page
3. Create a webhook endpoint in Shiplight using the copied URL
4. Run a test
5. View the received request on the webhook.site page

Local Testing (using ngrok)

# 1. Start local server
node your-webhook-server.js

# 2. Expose local port with ngrok
ngrok http 3000

# 3. Configure webhook with the HTTPS URL provided by ngrok

---

Best Practices

1. Security

• ✅ Always verify signature: Prevent forged requests
• ✅ Verify timestamp: Prevent replay attacks (recommend 5-minute tolerance)
• ✅ Use HTTPS: Protect data transmission
• ✅ Protect Secret: Store in environment variables, never hardcode

2. Performance

• ✅ Quick response: Return 200 response within 30 seconds
• ✅ Async processing: Put time-consuming operations in background queue
• ✅ Idempotency: Same webhook may be sent multiple times, ensure duplicate processing doesn't cause issues

3. Error Handling

• ✅ Return correct status codes:
  • 200: Successfully received
  • 401: Signature verification failed
  • 500: Server error
• ✅ Log errors: Facilitate troubleshooting
• ❌ No retries: Shiplight does not automatically retry failed webhooks (Phase 1)

4. Monitoring

Recommended monitoring metrics:

• Webhook reception success rate
• Processing time
• Error rate
• Signature verification failure count

---

FAQ

Q: Which HTTP methods are supported?

A: Only POST is supported.

Q: What is the timeout?

A: 30 seconds. Please ensure your server returns a response within 30 seconds.

Q: Are failed webhooks retried?

A: The current version (Phase 1) does not support automatic retries.

Q: Can I customize the payload format?

A: The current version uses a fixed format. Future versions may support custom templates.

Q: Can I configure multiple webhook endpoints?

A: Yes. You can create multiple endpoints, and each test plan/test suite can subscribe to multiple endpoints.

Q: Can I update the Secret?

A: Yes. Click "Edit" in Settings/Notifications/Webhooks to update the secret. Note: After updating, you need to synchronize the secret on your server.

Q: How do I test during local development?

A: Use ngrok or similar tools to expose your local service as an HTTPS address.

Q: Why does signature verification always fail?

A: Common reasons:

1. Using the wrong secret
2. Body was modified (make sure to use the raw body string, don't parse then stringify)
3. Timestamp exceeds 5-minute tolerance

---

Support

If you have questions, please contact technical support or check the API Documentation.