Test and Lint Gates That Really Stop

Why this matters

A lint gate fails on every third run because it downloads a tool version floating on latest, and developers learn to re-run until green. Then a real failure hides inside the noise and ships. Flaky gates train teams to distrust all gates; one unreliable check poisons the whole pipeline's authority. Gate reliability is a product feature of the pipeline, maintained like any other.

Concepts

Two gate classes with different contracts. Blocking gates reject the change: unit tests, type checks, lint with pinned rules, security scans on defined severity floors. Advisory signals inform without rejecting: coverage trends, performance deltas, experimental checks being evaluated. The failure mode is mixing them: an advisory wearing a blocking uniform stops releases for curiosity, and a blocking check demoted to advisory lets defects through. Write the classification down; review it when someone proposes a new check.

Determinism is the gate's job description. Pin every input: tool versions, dependency locks, base images, random seeds, clocks and network fixtures (use the local fake API discipline from M03). A gate that fails on Monday and passes on Tuesday with identical code is broken infrastructure, and the fix belongs in the gate definition, never in retry-until-green habits. Quarantine genuinely flaky tests out of the blocking set with an expiry date and an owner, or the suite rots around them.

Required checks close the human bypass. Branch protection (or the platform equivalent) names the checks that must pass before merge; without it, gates are advice the deadline overrules. Keep the required set small and fast so protection never feels like the enemy: under ten minutes for the blocking path is the budget most teams can defend. Everything slower moves to later stages or advisory signals.

Worked example

Unit tests pass locally and fail in CI every second run. The evidence:

FAIL src/orders.test.ts Expected: 2024-05-01T00:00:00.000Z, Received: 2024-05-01T02:00:00.000Z

Expected reading: the test asserts an absolute timestamp while CI runners span timezones, plus a network call to a sandbox API with no timeout from the M03 era. The repair pins TZ in the test environment, replaces wall-clock assertions with duration math, and stubs the network boundary. Verify with twenty consecutive green runs on a change that touches nothing, then promote the suite back to required. Lab L16's failing test is the mirror image: a real defect the gate must catch every time.

The common wrong move

Retry-until-green as policy, automated or cultural. Retries on genuine flakiness (network blips to the artifact store) are fine with limits and logging; retries as a way to launder red into green destroy the gate's meaning and hide the defect count. Count retries per gate per week: a rising line is a gate decaying, and the response is repair or quarantine, never acceptance.

Lab and next step

Lab L16 proves the blocking path end to end: introduce a real failure, watch the pipeline halt, read the record. Next, lesson 3 follows the artifact the gates certified: one version, one immutable artifact, honest caching.