Continuous testing in your CI/CD pipeline

Set the wait time for each pipeline stage

Before deciding what to run where, decide what developers are allowed to wait for. A starting budget teams can tune to their CI:

• Pre-push / commit: keep the loop short enough that people run it locally.
• Pull request: keep required checks within a review-friendly wall-clock budget.
• Deploy: keep smoke checks short because they block the release itself.
• Scheduled (production): unbounded runtime is fine — nobody is waiting.

The budgets are the strategy. Anything that can’t fit a stage’s budget moves right — to a later stage where waiting is acceptable — instead of slowing the earlier one. This is the core of the shift-left discipline we describe in shift-left testing for shipping teams: catch bugs early, but only pay for early detection where it’s cheap.

Commit stage: seconds, not minutes

What runs before code even reaches shared branches should be nearly free: lint, typecheck, unit tests, maybe a fast subset of integration tests against containers. The goal is not coverage — it’s a tight loop that catches the majority of mechanical mistakes.

# package.json — the local loop stays under ~90s
{
  "scripts": {
    "check": "npm run lint && npm run typecheck && vitest run --changed HEAD"
  }
}

Note --changed: running only tests affected by the diff keeps the loop honest as the suite grows. If your “fast” stage regularly exceeds its budget, the fix is almost always sharding, caching, or moving slow tests right — never asking developers to wait longer.

PR stage: critical-path E2E behind required checks

The pull request is where behavior verification belongs. Run end-to-end tests over the critical paths your change could plausibly affect — signup, login, checkout, the top three workflows by traffic — plus targeted regression tests derived from the diff.

Two mechanisms make this stage real rather than decorative. First, the checks must be required, so a red mark actually blocks merge — we walk through that configuration in required status checks as a merge gate. Second, flaky failures must be quarantined within hours, not weeks, because a gate that fails randomly gets admin-overridden into oblivion. The Playwright plumbing itself is covered step by step in Playwright in GitHub Actions.

# .github/workflows/pr-e2e.yml (excerpt)
jobs:
  critical-path:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright install --with-deps
      - run: npx playwright test e2e/critical --shard=${{ matrix.shard }}/4
        env:
          STRIPE_BASE_URL: http://localhost:12111   # clone, not sandbox
      - uses: actions/upload-artifact@v4
        if: failure()
        with: { name: traces, path: e2e/test-results }

Deploy stage: smoke, then promote

After deploy to an environment, run a smoke suite: one pass through each critical flow against the deployed artifact, using cloned third parties so it neither spends money nor sends email. Smoke tests assert “the release can serve users,” not “every feature works.” Five minutes, hard limit; anything deeper belongs to scheduled production testing.

Promote only on green. The value here isn’t catching logic bugs — those were caught at PR time — it’s catching deployment bugs: missing env vars, wrong migrations, CDN misconfig, the staging-parity gaps described in why staging lies to you.

Production stage: schedules, not releases

The last stage runs continuously against production with read-only or marked identities: synthetic logins, search, checkout up to (but not through) payment, API contract probes. This is the safety net for everything upstream can’t see — real data shapes, real CDN behavior, real vendor drift. We cover the design in synthetic monitoring for production; the broader cadence map is in our guide to regression testing strategies.

# Every 15 minutes, from three regions
- name: login → dashboard loads < 2s
- name: search returns seeded result
- name: api/v1/orders schema matches OpenAPI
# Alert routing: page only on 2 consecutive failures, same region

Attach artifacts so failures can be diagnosed

Attach the available trace, screenshots, requests, and service results to a failed check so the author can start investigating. Capture the details your test needs and make sure teammates can open them. Molar Guard and Trace support this workflow where the beta integration is configured; recording and publication depend on that setup.

How pipelines lose trust

• Budget creep. Each new feature adds “just one more” PR test until median check time doubles and engineers start merging on red. Re-audit against the budget quarterly and move tests right.
• Flake debt. Retries hide flaky tests instead of fixing them; the retry count becomes your real reliability metric. Track first-run pass rate per suite and quarantine anything below it.
• Orphaned stages. Deploy smoke keeps running for an app whose flows changed months ago, asserting a UI that no longer exists. Every scheduled stage needs an owner who edits its scenarios as the product changes.

None of these are tooling problems; all three are policy problems with tooling symptoms. Write the policy down — budgets, quarantine rules, stage owners — and revisit it whenever someone says “CI is slow again.”