core/architecture/microservices.md

Microservices Architecture: Boundaries, Communication, and Data Ownership

Named postmortems (Amazon Prime Video, Twilio Segment, Uber DOMA) of premature or over-decomposed microservices, with real numbers, plus concrete thresholds for when to split a monolith and when not to.

Assumes you already know bounded contexts, saga/outbox, choreography vs. orchestration, and the distributed-monolith anti-pattern by name. This document is the evidence file: what actually happened at companies that over-split, what they measured, and what numeric thresholds practitioners use instead of "it depends."

Named Postmortems

Amazon Prime Video - Video Quality Analysis, 2023

The VQA team's first version used a fully distributed, serverless pipeline (AWS Step Functions for orchestration, S3 as intermediate storage between Lambda stages, one state transition billed per second of video analyzed). It hit a hard scaling ceiling at ~5% of expected load - Step Functions state-transition limits and S3 as an inter-service handoff both became the bottleneck, and the state-transition billing model made cost scale with orchestration steps, not useful work. They rearchitected into a single process running on ECS, moving frame data in memory instead of through S3. Result: 90% reduction in infrastructure cost at the same throughput. Lesson for an agent to internalize: distributed-by-default is not a safety default - a hot-path pipeline with tight step-to-step data dependencies is exactly the shape that a monolith (or a single service) suits, and serverless orchestration cost scales with steps, not requests, which can invert the expected cost curve.

Twilio Segment - "Goodbye Microservices," 2018

Segment's inbound event pipeline had grown to over 140 per-destination microservices, one per third-party integration (~3 new services added per month at peak). Consequences, with numbers: 3 full-time engineers spent most of their time just keeping the system alive; the shared-library upgrade rate across the fleet was 32 improvements/year because every change meant redeploying a subset of 140 services; the full integration test suite took up to an hour; on-call was paged routinely for load spikes on individually under-provisioned low-traffic services. They consolidated all destinations into one service ("Centrifuge") behind a shared queue. After consolidation: a single engineer could deploy in minutes instead of coordinating 140+ deploys, shared-library improvements rose to 46/year, and the test suite dropped to milliseconds via a recording/replay harness. The service-per-integration boundary looked correct by DDD instincts (each destination is logically distinct) but was wrong operationally - the integrations shared the same lifecycle, same deploy cadence, and same team, so per-service isolation bought nothing and cost three engineers' full attention.

Uber - DOMA, published 2020

By 2018 Uber had grown to roughly 2,200 microservices. Diagnosing a single incident could require tracing through ~50 services owned by 12 different teams; a "simple" integration needed 10 touch points; services nominally independent still had to be deployed together in practice (a distributed monolith by this document's own definition). Uber's fix (Domain-Oriented Microservice Architecture) did not re-monolith - it grouped the 2,200 services into ~70 domains across 5 layers with single-entry gateways per domain. Measured effect: one early adopter's feature-integration time dropped from 3 days to 3 hours; onboarding time dropped 25-50% fleet-wide. Lesson: at large scale the fix for "too many services" is not always "fewer services," it's "a coarser unit of ownership and a contract layer above the services" - a middle ground between monolith and flat microservices.

Stack Overflow - the deliberate non-split

Stack Overflow runs its Q&A platform (~200 sites, ~1.3B monthly pageviews) as a monolith on 9 on-premises web servers, each sustaining ~450 peak requests/second at ~12% CPU utilization. This is not a legacy holdout - it is a standing, publicly documented counter-example to "you need microservices to scale": vertical scaling plus a well-optimized monolith handled roughly this company's entire traffic on single-digit server counts for years.

Shopify - modular monolith at scale, not microservices

Shopify Core is a single Ruby on Rails application (~2.8M lines of code, ~500K commits, 1,000+ contributors over a decade) that deliberately did not split into microservices as it grew. Instead it enforces internal modularity with Packwerk - each internal "pack" declares explicit public interfaces and dependency direction, checked automatically, inside one deployable. This is the modular-monolith pattern this whole document recommends as the default: get the bounded-context discipline without the network hop.

Concrete Thresholds Practitioners Actually Use

QuestionRule of thumbSource/basis
What team size should own one service?~6-10 engineers ("two-pizza team"), owning it end-to-endAmazon's two-pizza rule, widely cited via Conway's Law literature
When does incident diagnosis prove the boundary is wrong?If root-causing a single incident requires touching 10+ services or 5+ teamsUber's pre-DOMA state (~50 services/12 teams per incident) was the trigger for restructuring
When does a "distinct domain" not deserve its own service?When every instance of that domain type shares deploy cadence, on-call, and team with its siblingsSegment: 140 destinations, one lifecycle - should have been one service from the start
When is serverless/distributed orchestration the wrong default?When steps have tight data dependencies and per-step billing scales faster than throughputAmazon Prime Video Step Functions/S3 bottleneck
When do you need a coarser layer instead of fewer services?Service count in the thousands, with cross-cutting features requiring 10+ touch pointsUber DOMA (2,200 services -> 70 domains)
When is a monolith still fine at real scale?Vertical scaling covers current peak RPS with headroom, and the team is small enough to coordinate one deploy pipelineStack Overflow (9 servers, ~1.3B pageviews/month), Shopify (single Rails monolith, modularized internally)

Common Pitfalls & Anti-Patterns (Reinforced by the Cases Above)

The Distributed Monolith

Services split by technical convenience or org chart, not bounded context, that must still deploy together or share a database. Audit directly: if two services cannot be deployed independently without breaking each other, they are not microservices yet. Uber's pre-DOMA fleet exhibited this at 2,200-service scale; it is equally possible at 5.

Shared Database Between Services

Any schema change now requires coordinating multiple teams' releases - this alone reintroduces the distributed monolith regardless of how many deployable artifacts exist.

Premature Decomposition

Splitting a brand-new product into many services before the domain model or team structure has stabilized front-loads the full operational tax (deploy pipelines, tracing, service discovery, retries/circuit breakers) before there is any organizational benefit to collect. Segment's 140 per-destination services is the textbook example: correct-looking DDD boundary, wrong operational boundary, 3 engineers' full-time cost to maintain.

Assuming Serverless/Managed Orchestration Removes This Trade-off

It doesn't - it just moves the cost from engineering time to a metered bill that can scale worse than a monolith under the wrong access pattern (Amazon Prime Video's Step Functions state-transition billing).

Testing Strategies

  • Unit-test business logic within each service exactly as in a monolith.
  • Contract-test every service-to-service boundary (Pact-style, consumer
  • publishes expectations, provider verifies in its own CI before deploy).

  • Avoid broad end-to-end tests spanning all services as the primary safety
  • net - slow, flaky, and this is exactly the failure mode that made Segment's pre-consolidation suite take up to an hour.

  • Test saga compensations explicitly: force failure at each step and
  • assert the correct compensating actions fire in order.

Performance, Security & Scalability Considerations

  • Budget and measure end-to-end latency across the whole call chain, not
  • per-service - Uber's 50-service incident chains show how this compounds.

  • Apply timeouts, retries with jitter, and circuit breakers on every
  • synchronous inter-service call.

  • Authenticate and authorize service-to-service calls (mTLS or signed
  • tokens) as rigorously as external-facing calls - do not assume same-cluster trust.

  • The scaling win only materializes if the boundary actually isolates the
  • bottleneck component; if it doesn't, you pay network/ops cost for zero scaling benefit (Prime Video's original design).

When Not to Use / Alternatives

DimensionMicroservicesModular MonolithMonolith (unstructured)
Independent deployabilityExcellent, if boundaries are rightNone (single deploy unit)None
Operational complexityHighLowLowest
Demonstrated failure mode at scaleSegment (140 services, 3 FTE to sustain), Uber (2,200 services, 10-touchpoint integrations)N/AN/A
Demonstrated success at scale without splittingN/AShopify (2.8M LOC Rails monolith, Packwerk-enforced modules)Stack Overflow (9 servers, 1.3B pageviews/mo)
Best fitLarge orgs, clearly separable domains, differing scaling/tech needs, proven by production pressureMost products, especially pre-product-market-fitSmall, short-lived projects

Default to a modular monolith. Split out a specific module only when you have a demonstrated, currently-occurring need: independent scaling (a component is provably the bottleneck), independent deploy cadence across teams, or a genuinely different technology/runtime requirement. Every named regret case above (Prime Video, Segment, pre-DOMA Uber) split before that pressure existed and paid an operational tax to discover it - then partially or fully reversed the split.

Sources

  • Amazon Prime Video Tech Blog, "Scaling up the Prime Video audio/video monitoring service and reducing costs by 90%" - https://www.primevideotech.com/video-encoding/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90
  • The New Stack, "Return of the Monolith: Amazon Dumps Microservices for Video Monitoring" - https://thenewstack.io/return-of-the-monolith-amazon-dumps-microservices-for-video-monitoring/
  • Twilio/Segment Engineering, "Goodbye Microservices: From 100s of problem children to 1 superstar" - https://www.twilio.com/en-us/blog/developers/best-practices/goodbye-microservices/
  • InfoQ, "Why Segment Returned to a Monolith from Microservices" - https://www.infoq.com/news/2018/07/segment-microservices/
  • Uber Engineering Blog, "Introducing Domain-Oriented Microservice Architecture" - https://www.uber.com/en-SE/blog/microservice-architecture/
  • Stack Overflow / Nick Craver, architecture write-ups summarized - https://newsletter.techworld-with-milan.com/p/stack-overflow-architecture
  • Shopify Engineering, "Under Deconstruction: The State of Shopify's Monolith" - https://shopify.engineering/shopify-monolith
  • Shopify Engineering, "Deconstructing the Monolith" - https://shopify.engineering/deconstructing-monolith-designing-software-maximizes-developer-productivity
  • Martin Fowler, "MonolithFirst" - https://martinfowler.com/bliki/MonolithFirst.html