Flaky tests: how to detect, measure and fix them
A flaky test passes and fails without a relevant change. A simple formula to measure flakiness from test history, common root causes and a triage process that works.
A flaky test is one that passes and fails on the same code without a relevant change. Measure flakiness from history: count how often a test's outcome flips between pass and fail across its recent decisive results, divide by the number of decisive results minus one, and investigate tests that score above about 0.3.
Why flaky tests are expensive
A flaky test costs more than the minutes spent re-running it. Each false failure teaches the team to ignore red results, and once failures are ignored, real regressions slip through. Flakiness is a trust problem first and a time problem second.
A simple way to measure flakiness
You do not need machine learning to find flaky tests; your result history is enough.
- Take the test’s recent decisive results, pass or fail, ignoring skipped and blocked.
- Require at least three decisive results that include both a pass and a fail.
- Count the flips: each time the outcome changes from one result to the next.
- Divide by the number of decisive results minus one.
Flaky score = flips ÷ (decisive results − 1)
| History (oldest → newest) | Flips | Score | Reading |
|---|---|---|---|
| P P P P P F | 1 | 0.20 | Probably a real regression |
| P F P P F P | 4 | 0.80 | Very flaky |
| P P F F F F | 1 | 0.20 | Broke and stayed broken |
| P F P F | 3 | 1.00 | Alternates every run |
A test that fails once and stays failing has a low score, which is what you want: that is a regression, not flakiness. Tests scoring around 0.3 or more deserve investigation. TestPlum uses this exact formula in its stability report.
Common root causes
- Timing: fixed sleeps, missing waits for asynchronous work, animations.
- Shared state: tests that depend on data another test created or deleted.
- Order dependence: a test passes only after another test has run.
- External services: network calls to systems you do not control.
- Dates and time zones: tests that fail near midnight, at month end or in another region.
- Resources: CPU or memory contention on busy CI machines.
- Randomness: unseeded random data that occasionally hits an edge case.
A triage process that works
- Detect with the score above, every day, from real results.
- Rank by score and by priority; a flaky critical-path test matters more.
- Quarantine the worst offenders so they stop blocking pipelines, while still running and recording them.
- Fix the root cause, then watch the score fall over the next runs.
- Prevent: review new tests for the root causes above.
Retries: useful, with a condition
Automatic retries keep pipelines green, but if only the final attempt is recorded, flakiness becomes invisible. Record every attempt so the history, and therefore the score, stays honest.
Flakiness and release decisions
A flaky failure should not block a release on its own, but it should not be waved through either. Re-run it, look at its history, and decide with evidence. See the release readiness checklist and how TestPlum reports stability and flaky tests.
Frequently asked questions
What causes flaky tests?
The most common causes are timing and asynchronous waits, shared or leaking test data, order dependence between tests, unreliable external services, time zones and dates, and resource limits on CI machines.
Should you retry flaky tests automatically?
Retries hide the symptom and keep pipelines green, but they also hide real intermittent bugs. If you retry, record every attempt so flakiness stays measurable, and fix or quarantine the worst offenders.
How many results do you need to call a test flaky?
At least three decisive (pass or fail) results that include both outcomes. With fewer results a single failure cannot be told apart from a real regression.