How self-healing works

View as Markdown

Shiplight repairs tests at two levels. Step-level self-healing occurs automatically during a run: when a cached locator becomes stale, the AI reads the page again, resolves the step's intent, and continues the test. Test-level repair uses /shiplight fix: your coding agent reproduces the failure, identifies the cause, and updates the YAML or reports an application bug. This guide explains when to use each type of repair.

Two levels of repair

	Step-level self-healing	Test-level /shiplight fix
When it runs	Automatically during a test run	When you or CI auto-triage starts it
What it fixes	A stale cached locator for one step	Locator changes, behavior changes, test bugs, expired authentication, or invalid fixtures
What changes	The run continues and the cache is updated	The YAML in your repository through a standard commit
Who acts	The Shiplight runtime	Your coding agent

Step level: heal mid-run

Enriched ACTION statements include a cached action: and locator: so Shiplight can replay them deterministically in less than a second without calling a model. The cache improves performance, but the step's intent: remains the source of truth for the expected action.

When a cached locator fails because a button was renamed, the DOM structure changed, or a class changed, Shiplight performs the following recovery process:

1. The runtime catches the locator failure.
2. The AI reads the current page and resolves the step's intent: against the available elements.
3. It performs the resolved action, and the run continues.
4. Shiplight captures the new locator. After the run passes, it saves the locator as a cache update. On Shiplight CI runners, Shiplight writes healed locators to Shiplight Cloud so later runs can use the updated cache.

Shiplight does not rewrite the YAML in your repository during a run. The healed locator remains in the cache until your coding agent commits an update to the file.

What self-healing costs

A healed step calls the model the first time it resolves a new locator, so that step takes several seconds instead of replaying from the cache in less than a second. In CI, the shared Shiplight Cloud action cache can reuse the healed locator on later runs and avoid most repeated model calls. Local runs do not share this cache. If the locator change is permanent, update the test so its YAML contains the current cached action: and locator:. Run /shiplight fix, or let CI auto-triage open a pull request, to make that update.

What can and cannot heal

• ACTION statements with a locator can heal because the natural-language intent: gives the AI enough information to resolve the step again.
• VERIFY statements use AI assertions on every run. A js: cache on a VERIFY only improves performance. If the expression throws an error, the runtime uses the natural-language statement for AI verification.
• Raw code does not heal. Code steps (description: + js:), custom functions, and plain waits do not have an intent: to use as a fallback. If the code throws an error, the step fails. Use raw JavaScript only when necessary, and prefer intent: with action: for other steps.

Test level: /shiplight fix

Self-healing allows a run to continue after mechanical locator changes. For other failures, /shiplight fix reproduces the problem, determines whether it was caused by locator drift, an intended behavior change, a test bug, or a coverage gap, and applies the smallest correct update to the YAML. If the application is broken, the agent reports the bug instead of changing the test to pass.

Fix a failing test describes the complete diagnostic process and its safeguards. In CI, the auto-triage pipeline can perform the same repair automatically after a failure. It opens a pull request but does not merge it.

Choose the appropriate repair

• No action required: A locator changed once and healed successfully.
• Use /shiplight fix: A test is failing, or a permanent UI change should be reflected in the cached action: and locator: in YAML.
• Plan a rewrite instead of a repair: The product is being intentionally redesigned. Reassess the test plan with Decide what to test, then create tests for the new workflow.

Related

• YAML Test Format overview: Learn how test enrichment works and why Shiplight treats locators as a cache.
• Statement Types: Review ACTION, VERIFY, DRAFT, and code steps.
• Fix a failing test: Follow the test-level repair workflow.
• CI/CD: Configure auto-triage and the hosted action cache.