Choose which regression tests to run before release

Why full-suite gating stops scaling

Regression suites fail in a predictable way as teams grow. At five engineers, “run everything on every push” works — a 10-minute suite is invisible. At fifty engineers with a 90-minute E2E suite, the math stops working: ten concurrent PRs contend for runners, feedback latency kills flow state, and engineers start batching changes to “make CI worth it,” which multiplies both merge conflicts and risk.

The standard responses are both wrong. Cutting the suite (“we’ll rely on code review”) trades known costs for unknown ones — production incidents that a 30-second check would have caught. And brute-forcing with more machines hits an economic wall: E2E suites grow superlinearly in runtime because flows share setup and interact. The exit is not more hardware or fewer tests. It’s running the right tests, at the right time, with full knowledge of what changed.

Risk-based selection starts from the diff

A pull request is a hypothesis about what it affects. Selection should encode exactly that hypothesis: map files to modules to user flows, then run the tests that exercise the touched surface plus everything downstream of it.

# Illustrative selection policy — adapt to your CI tooling
rules:
  - touch: "src/checkout/**"
    run_flows: [cart, checkout, order-confirmation, refunds]
  - touch: ["src/auth/**"]
    run_flows: [signup, login, password-reset, session-security]
  - touch: "package-lock.json"
    run_flows: all            # dependency bumps get everything
always_run: [smoke]           # never skipped

The payoff is concrete: a PR that edits invoice rendering runs only the affected flows, while a dependency bump still gets the world. Keep the mapping in the repo, reviewed like code, because it is code — it encodes your beliefs about coupling, and stale mappings rot into silent gaps. This is also where AI-authored suites change the economics: when test generation is cheap, covering every affected flow stops being a headcount problem.

Map code changes to the tasks they can affect

File-path rules are the floor, not the ceiling. A change to a shared component or an API response shape touches flows its path never mentions. The stronger version builds an actual dependency graph: which routes render this component, which flows traverse those routes, which API endpoints feed them.

Example mapping to review:
PricingTable.tsx changes
→ Pricing and upgrade pages use this component
→ Run plan-selection and checkout tests
→ Include the normal smoke checks

Two failure modes to design against. Over-selection (everything depends on everything) usually means a shared utility needs decoupling — treat it as architecture signal, not a tool bug. Under-selection is the dangerous one, and you catch it with data: when a nightly run finds a regression your selection missed, trace back which edge was absent from the graph and add it. Escape analysis feeds selection accuracy; see the measurement section below.

The nightly full suite is insurance, not gating

Even good selection has blind spots: interactions between flows, environment drift, vendor behavior changes, cache poisoning that only manifests after hours of activity. The nightly full run exists to catch exactly those. Treat it as insurance — nobody waits on it to merge — but take its failures seriously: anything red at night that was green on merge is either a real cross-flow regression or a flaky test, and triage decides which within the morning.

• Schedule it off-peak against production-like data, using clones for third-party side effects so the run is safe to repeat (no dedicated staging needed).
• Capture evidence per flow — traces, network logs, DOM snapshots — so morning triage is reading, not re-running.
• Track suite duration weekly. A nightly that grows steadily is telling you something about test debt before the PR gate feels it.

Production schedules extend the same idea outward: Guard-style recurring checks against live environments catch what no pre-merge suite can — CDN issues, cert expiry, vendor drift. That’s the complement to synthetic monitoring, not a replacement for the nightly.

Quarantine flaky checks with an owner

Flaky tests are where regression programs die. Not because flakes are unfixable — most have diagnosable causes (root causes and fixes here) — but because teams lack a shared policy, so every engineer improvises: mute it, delete it, retry-until-green, ignore red. Write the policy down instead:

• Quarantine means tracked. A quarantined test moves out of the gate, gets an owner and a ticket, and shows up on a dashboard. It does not vanish.
• Time-boxed hard. Fourteen days default. Quarantine without expiry is how suites rot while dashboards stay green.
• Snooze ≠ quarantine. Snoozing a known-flaky test during a feature branch is fine; quarantining a failing test to ship a release requires the owner’s name next to it.
• Budget the rate. If a substantial share of the suite is quarantined at steady state, stop merging features and fix reliability — the gate’s credibility is the product.

Review regressions your tests missed

The only number that tells you whether any of this works is escape rate: regressions that reached users divided by total regressions found, over time. Everything else — coverage percent, test count, suite runtime — is an input metric that can look great while customers find bugs first.

escape_rate = bugs_found_in_production_with_regression_origin
            / (same_numerator + regressions_caught_by_suite)

# Track monthly, segment by area, and review every escape
# Every escape → write the missing test + trace why selection missed it

Track time to useful feedback and the time between a production failure and a response. Review escaped regressions to learn whether a test was missing, an assertion was weak, or a selection rule skipped the right check. Adjust the rule or test based on that finding.

Finally, make the gate itself non-negotiable: required status checks turn “please run the tests” into physics. The mechanics are in our guide to required checks as a merge gate.