core/testing/unit_testing_principles.md

Unit Testing Principles: FIRST, the Test Pyramid, and What Makes a Test Good

The pyramid-vs-trophy debate (Fowler/Google vs. Kent C. Dodds) with attribution, Google's small/medium/large size taxonomy and 80/15/5 split, the Luo et al. flaky-test root-cause taxonomy with real percentages, and mutation testing as the actual suite-quality signal coverage isn't.

unit-testingtestingtddtest-pyramidtesting-trophyflaky-testsmutation-testingbest-practicespytest

Assume the reader already knows FIRST, Arrange-Act-Assert, and what a unit test is. This document covers what isn't textbook material: the named pyramid-vs-trophy debate, Google's actual internal ratios and vocabulary, the sourced flaky-test taxonomy, and why coverage percentage is not a suite-quality metric.

The Pyramid-vs-Trophy Debate: Two Named, Conflicting Positions

Position A - the classic pyramid (Mike Cohn, popularized further by Martin Fowler and Google's Testing Blog): many fast unit tests, fewer integration tests, very few end-to-end tests, ordered by cost/speed/flakiness. Google's own internal guidance ("Software Engineering at Google," ch. 11) states a concrete target ratio of 80% unit / 15% integration / 5% end-to-end, and its 2015 Testing Blog post "Just Say No to More End-to-End Tests" (Mike Wacker) argues E2E suites should be used sparingly because they are unstable and slow to diagnose.

Position B - the testing trophy (Kent C. Dodds, kentcdodds.com/blog/write-tests): static analysis (TypeScript/ESLint-class tooling) as a free base layer, then integration tests as the largest, most-invested layer, with unit and E2E both smaller. Dodds' stated reasoning is a confidence/cost trade-off, not a speed argument: "as you move up the pyramid, the confidence quotient of each form of testing increases," and integration tests get most of that confidence gain without full E2E cost. His companion aphorism, widely quoted verbatim: "Write tests. Not too many. Mostly integration."

These are not the same claim answered differently - they optimize for different failure modes. The pyramid optimizes for feedback speed and failure localization (Google's own codebase is enormous and refactored constantly by thousands of engineers; a unit-heavy suite gives fast, precise blame). The trophy optimizes for confidence-per-test in smaller, UI-heavy codebases where unit tests of individual functions frequently pass while the wired-together feature is broken (the "unit tests mocked the seam that was actually broken" failure - see Mocking Strategies).

Decision rule:

Codebase shapeLean toward
Large monorepo, many independent pure-logic modules, frequent large-scale refactorsPyramid (Google ratio)
Small-to-mid app where most bugs live at component/module wiring boundaries (frontend, API-glue-heavy backends)Trophy (integration-heavy)
Either - but end-to-end suite is the only safety netNeither is satisfied; add unit and/or integration tests, don't add more E2E (Wacker's core argument)

Google's Size Taxonomy Is a Better Vocabulary Than "Unit/Integration/E2E"

Google replaced "unit/integration/e2e" internally with small / medium / large / enormous, scoped by what the test is allowed to touch, not by what layer of the architecture it happens to exercise:

  • Small: single process, no network, no sleeping, no blocking I/O - must run in milliseconds.
  • Medium: can span processes on one machine, can hit localhost, can touch disk.
  • Large/Enormous: multi-machine, real network, real external services.

This taxonomy is more decision-useful than the pyramid's layer names because it's enforceable by CI infrastructure directly (a test tagged small that opens a socket is a build error at Google, not a code-review nitpick) rather than relying on a human classifying which "layer" a test conceptually belongs to.

Flaky Tests: A Sourced Root-Cause Taxonomy, Not a Mystery

Luo, Hariri, Eloussi, and Marinov, "An Empirical Analysis of Flaky Tests" (FSE 2014; mir.cs.illinois.edu/lamyaa/publications/fse14.pdf) mined 201 flakiness-fixing commits across 51 open-source Java projects and produced the taxonomy most CI tooling still cites in 2026:

Root causeShare of studied flakesFix pattern
Async wait (fixed sleep() instead of polling/condition)~45%Poll a condition with timeout; never bare-sleep. Fowler's independent guidance (martinfowler.com/articles/nonDeterminism.html) reaches the identical conclusion.
Concurrency (race conditions, atomicity violations, deadlocks)~20%Eliminate shared mutable state between tests/threads; make the ordering deterministic under test.
Test-order dependency~12%Function-scoped fixtures, no test writes to state another test reads; shuffle test order in CI (pytest-randomly) to surface this class proactively.
Resource leakremainder, recurring categoryConfigure test resource pools small enough that a leak fails fast and loud instead of silently accumulating.
Network / external dependencyrecurring categoryPush out of the unit layer entirely - this is what the integration layer and Testcontainers exist for.
Time dependencyrecurring categoryInject a Clock port; never call the system clock directly from logic under test.

Google's own flaky-test writeup (testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html) independently confirms the same top causes at their scale (UI-harness timing, poor test isolation via shared global/temp-file state, unreliable external connections) and adds an operational pattern worth stealing directly: a quarantine system that reruns newly-flaky tests automatically, strips them out of the blocking critical path, and enforces a hard time limit before a quarantined test must be fixed or deleted - flaky tests left both flaky and blocking are what trains engineers to ignore red CI.

Decision rule for triage: when a test fails intermittently, check causes in the above order - async wait is the single most likely explanation before you assume something exotic. Never mark a test @retry/quarantine without opening a tracked ticket; a permanently-retried test is a silently disabled test.

Mutation Testing Is the Actual Suite-Quality Signal - Coverage Is Not

Line/branch coverage measures whether code executed during the suite, not whether a bug in that code would have been caught. A test with assert result is not None on a line that computes a completely wrong value still counts as "covered." Mutation testing (mutmut, cosmic-ray for Python; Stryker for JS/TS) tools deliberately inject small bugs (flip a comparison operator, off-by-one a boundary) and measure the kill rate - the fraction of injected mutants that make some test fail. A suite with 100% line coverage and a low mutation-kill rate is testing that code ran, not that it's correct - treat a coverage number alone as a floor to find untested branches, never as a suite-quality target.

When Not to Invest Further in Unit Tests

SignalImplication
Bugs keep reaching production despite green, high-coverage unit suiteThe bugs live at wiring/integration boundaries the unit layer cannot see by construction - invest in the trophy's integration layer instead, don't add more unit tests of the same units.
A 5-line change breaks dozens of unit testsBrittle, implementation-coupled suite (see Mocking Strategies) - fix the tests' coupling, don't treat the breakage count as validation.
Suite takes minutes locallySomething in the "unit" layer is doing real I/O; reclassify it as medium/integration per Google's taxonomy above and move it to that CI stage.

Sources