Guides
Repair flaky tests without weakening them
Reproduce intermittent failures within a declared budget, fix demonstrated causes, and report unreproduced suspects honestly.
Answers: use a coding agent to repair flaky tests · Updated
A flaky-test shift begins with existing instability evidence and a repetition budget. It fixes deterministic causes such as leaked state, ordering, time, randomness, concurrency, environment, or resources—never the symptom through retries, skips, muted assertions, or arbitrary timeouts.
Require evidence and a repetition budget
Collect named suspects from CI history, failure artifacts, repeated local failures, or an owner-provided list. Declare how many repetitions each suspect receives before work starts.
- Use the repository’s existing test runner.
- Reproduce one suspect at a time.
- Record commands, run count, and outcome.
Fix only a demonstrated cause
When the failure reproduces, isolate the deterministic cause, repair it, run the project gate, then repeat the repaired test for the same budget.
- Do not add retries as the repair.
- Do not skip, quarantine, mute, or weaken the test.
- Do not increase a timeout unless the timeout itself is the proven contract.
Unreproduced is a valid result
If the suspect never fails inside the declared budget, record it as unreproduced and move on. The shift does not claim a repair it cannot demonstrate.
Run the flaky-test repair ready shift with a declared repetition budget. Fix reproduced causes and record unreproduced suspects without claiming success.Start with a bounded prompt
This prompt names the outcome and preserves Nightshift’s review boundary. Paste it into the supported coding host from the project you want to change.
Run the flaky-test repair shift with a declared repetition budget. Repair only reproduced causes and record unreproduced suspects without claiming a fix.What this workflow does not claim
- Retries, longer timeouts, skipped tests, and weaker assertions are not accepted as repairs without contract evidence.
- The shift needs existing instability evidence and the project's existing test runner.
Evidence and sources
These links support the released behavior, public outcomes, or problem language described on this page.