Treat intermittent failures as bugs to investigate

When the same commit passes on one run and fails on the next, teams learn the worst possible habit: re-run until green. Once that habit forms, the suite stops being a gate and becomes weather — people check it, nobody trusts it. The economic damage is quiet but compounding: engineers buffer their estimates for “CI roulette”, real regressions hide among false ones, and eventually someone proposes deleting the suite entirely.

The useful reframe is that flakiness is not random. In our experience, most intermittent failures fall into one of five groups: timing races, shared state, environment drift, third-party failure, and selector brittleness. Each group has a durable fix. Find the group first; reaching for retry before diagnosis just moves the problem from your failure report into your blind spot.

FAILED ASSERTION

Expected state missing

Keep the original assertion visible; do not replace it with “page loaded.”

start here
REPRODUCE

Same account, same input

Repeat from a clean state and separate product failure from setup failure.

control variables
INSPECT

Browser · request · console

Align the browser, request, and console records around the first divergence.

find the first break
NEXT STEP

Run one new experiment

Change one variable, then decide whether the fix belongs in code or the test.

reviewed diagnosis
Move from a red check to a causeA useful failed run preserves the failing assertion, the relevant request, and the next experiment.

Timing races: sleeps are guesses

The classic flake: the test works on your fast laptop, fails in CI where eight containers share four cores. The app takes 400ms to render instead of 200ms, and a fixed sleep that was tuned to “usually work” now loses the race. Any test that waits a duration is really waiting for luck.

Code example
// Fragile: sleeps guess at timing
await page.waitForTimeout(3000);
await page.click('#submit');

// Durable: act, then wait on observable state
await page.getByRole('button', { name: 'Pay now' }).click();
await expect(page.getByText('Payment confirmed')).toBeVisible();

Web-first assertions flip the model: instead of sleeping then checking once, they poll observable state until it’s true or a timeout expires. Two companion rules close the gap entirely. First, treat unhandled promise rejections and unexpected network errors as test failures rather than console noise — they are tomorrow’s flakes announcing themselves today. Second, freeze sources of variance you don’t need: animations off, fake clocks for anything time-based, deterministic IDs for anything paginated.

Shared state: two tests, one row

The second bucket shows up when tests collide through something they both touch: the same seeded user, the same database table, the same localStorage profile. Symptoms include assertions on counts (“cart should have 3 items” seeing 7), duplicate-key errors that appear only under parallelism, and tests that pass alone but fail in the full run. Parallel execution doesn’t create these bugs — it reveals them.

Durable fixes, in order of preference: give every test its own data (factories that mint unique users and orders per invocation, then clean up); scope expensive shared fixtures per worker rather than globally; and where isolation must be absolute, roll back via transactions or run each worker against its own schema or container. If a test needs a specific global to exist, that dependency belongs in its setup — explicit, owned, and visible — never in the residue of whichever test ran last.

Environment drift: it passed on my machine

Some flakes aren’t in your code at all. The staging server was mid-deploy. A feature flag flipped. CI runs in a region where the page renders in another locale, or under a timezone that makes your “end of month” invoice test compute a different month. These failures look mysterious precisely because the cause lives outside the test.

Countermeasures are organizational more than technical. Pin everything your tests implicitly depend on: timezone, locale, feature-flag state, seed data version. Prefer short-lived preview environments created per pull request over a long-lived shared staging box that accumulates entropy — see our guide to staging environment alternatives. And add a pre-flight health check so a half-deployed app produces “environment unhealthy”, not forty misleading test failures.

Third-party failures: someone else’s uptime

Checkout tests that hit a live payment sandbox inherit its outage schedule. Email tests wait on real SMTP relay latency. Rate limits turn a burst of parallel tests into intermittent 429s. None of this is your product breaking, yet your build goes red and your team burns an afternoon proving it.

The durable fix is to stop sharing fate with third parties during routine runs. Stateful substitutes that speak the vendor’s API — returning deterministic IDs, replaying webhooks, delivering email to an inspectable inbox — remove the network, the latency, and the outages while exercising your real code paths. We compare the options in API mocking versus service clones, and walk a payments example in testing Stripe without real cards. Keep exactly one scheduled smoke against the real sandbox as a drift alarm, and you get hermetic daily runs without blinding yourself to contract changes.

Replace selectors tied to page structure

The fifth bucket degrades gradually rather than intermittently, but it manufactures flakes all the same: a selector like div:nth-child(3) > span.btn-primary survives until a designer wraps one node in a new div, and suddenly the locator matches nothing — or worse, matches a different button. Ambiguous text selectors produce the intermittent flavor: they pass while unique, then silently bind to a newly added element.

The fix ladder is well established: prefer role plus accessible name (getByRole('button', { name: 'Checkout' })), then label associations, then purpose-built data-testid, then unique text, and treat structural CSS as a code smell. When the UI legitimately changes, a disciplined self-healing layer can re-rank locators and propose the update for review instead of failing the build — provided intent was captured at authoring time.

Set a retry limit and assign an owner

Retries deserve nuance. Used deliberately — one retry, both outcomes reported — they absorb genuine environmental noise and generate a flake-rate metric you can manage down. Used silently, they convert your suite into a lottery where the prize is ignorance. The difference is whether a retry that flips red to green triggers investigation or celebration.

A retry that passes is a bug report, not a pass. If your green build contains three retried tests, you shipped with three undiagnosed defects wearing a green badge.

Set a small retry limit and report when a run needs it. Give repeated flakes an owner and a repair deadline; keep them visible if they temporarily stop blocking merges. A retry should not erase the first failure. See the GitHub Actions guide for running a configured Molar scenario in CI.