---
title: "Integration Testing: Real Boundaries, Ephemeral Infrastructure, and Contract Testing"
description: "How to test across real system boundaries - databases, HTTP APIs, and message queues - using ephemeral test infrastructure (Testcontainers) and consumer-driven contract testing (Pact-style), with an honest accounting of trade-offs against pure unit tests."
language: null
framework: null
category: testing_general
tags:
  - integration-testing
  - testing
  - testcontainers
  - contract-testing
  - pact
  - microservices
  - database-testing
keywords:
  - integration testing testcontainers python
  - contract testing pact consumer driven
  - testing real database vs mocking database
  - ephemeral test infrastructure ci cd
  - integration testing message queues kafka
last_updated: 2026-07-07
difficulty: intermediate
version: "Testing theory; examples in Python/pytest with testcontainers-python"
related:
  - unit_testing_principles.md
  - mocking_strategies.md
  - ../architecture/microservices.md
  - ../architecture/hexagonal.md
  - ../devops/ci_cd_patterns.md
  - ../../languages/python/frameworks/fastapi/best_practices.md
search_priority: high
status: published
---

# Integration Testing: Real Boundaries, Ephemeral Infrastructure, and Contract Testing

## Introduction / Overview

Integration testing verifies that a component behaves correctly when combined with a **real** dependency it does not control end-to-end - a database, an HTTP API, a message broker, or a file system. Where unit tests deliberately isolate the code under test from these dependencies (see [Unit Testing Principles](unit_testing_principles.md)), integration tests deliberately reintroduce exactly one such dependency to verify the assumptions the unit tests had to fake.

The central engineering problem integration testing solves is: a repository class can be perfectly unit-tested against an in-memory fake and still be entirely wrong, because the fake encodes the developer's *assumptions* about how the real database behaves, not the database's actual behavior. Integration tests exist to validate that assumption against the real thing, in an environment fast and disposable enough to run routinely.

By 2026, the default way to provide that "real thing" in CI and locally is **ephemeral test infrastructure** - Testcontainers-style libraries that programmatically start a real, disposable instance of a dependency (Postgres, Kafka, Redis) in a container for the duration of a test run and tear it down afterward, rather than relying on a shared, long-lived test environment.

**When to use this guidance**: Testing repository/adapter implementations, HTTP client integrations, message consumers/producers, and any code whose correctness depends on the real behavior of an external system rather than on the calling code's own logic.

## Core Concepts

- **Boundary**: Any point where your code hands control to, or receives control from, a system you do not fully control the internals of - a database engine, another service's API, a message broker, the file system, the system clock.
- **Ephemeral Test Infrastructure**: Real dependencies started fresh for a test run (typically via Docker) and destroyed afterward, giving integration tests the realism of production infrastructure with the repeatability of a unit test - no shared, stateful "test database" that different test runs can pollute.
- **Testcontainers**: A library family (Java-originated, with mature Python, Go, .NET, and Node ports) providing a programmatic API to start throwaway, versioned containers (Postgres, Kafka, LocalStack, etc.) from test code, with built-in wait strategies so a test does not proceed until the dependency is actually ready.
- **Contract Testing**: A technique for verifying that two independently deployed services agree on the shape of their communication (a request/response schema, an event schema) **without** running both services together. A consumer records its expectations as a contract; the provider verifies, in its own CI pipeline, that it satisfies every recorded consumer contract.
- **Consumer-Driven Contracts (Pact-style)**: The consumer team owns and authors the contract (from their real usage), publishes it to a shared broker, and the provider team's CI fetches and verifies against it - inverting the traditional "provider defines the API spec" flow so the contract reflects actual consumer needs.
- **The Integration Test Trade-off**: Integration tests are slower and more failure-prone than unit tests (they depend on Docker, network, and the target system's startup time) but they catch an entire category of bugs - wrong SQL, wrong serialization format, wrong HTTP status-code handling, wrong message schema - that no amount of unit testing against fakes can catch, because the fake's correctness is exactly what is in question.

## Best Practices

### 1. Test One Real Boundary per Integration Test, With Everything Else Faked

**Rationale**: Mirroring [Hexagonal Architecture](../architecture/hexagonal.md), an integration test for the `SqlAlchemyOrderRepository` should use a real Postgres container but does not need a real email service - fake collaborators that are not the boundary under test.

```python
import pytest
from testcontainers.postgres import PostgresContainer
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession

@pytest.fixture(scope="session")
def postgres_container():
    with PostgresContainer("postgres:16-alpine") as container:
        yield container

@pytest.fixture
async def session(postgres_container):
    engine = create_async_engine(postgres_container.get_connection_url())
    async with AsyncSession(engine) as session:
        yield session
    await engine.dispose()

async def test_repository_persists_and_retrieves_order(session):
    repo = SqlAlchemyOrderRepository(session)
    order = Order.create(items=[LineItem(price=Decimal("10.00"))])

    await repo.add(order)
    fetched = await repo.find_by_id(order.id)

    assert fetched is not None
    assert fetched.total == Decimal("10.00")
```

### 2. Use Real Wait Strategies, Never `sleep()`

**Rationale**: A fixed `sleep(5)` is both slow (wastes 5 seconds every run) and fragile (5 seconds is not always enough under CI load). Testcontainers-family libraries expose log-based or port-based wait strategies that block exactly until the dependency reports readiness.

```python
from testcontainers.kafka import KafkaContainer

with KafkaContainer() as kafka:
    bootstrap_servers = kafka.get_bootstrap_server()  # blocks until broker is ready
```

### 3. Pin Container Versions to Match Production

**Rationale**: Using `postgres:latest` means the test environment silently drifts from production as new images are published, occasionally introducing incompatibilities that only surface in CI. Pin to the exact major/minor version running in production.

### 4. Never Enable Container Reuse in CI

**Rationale**: Container reuse (keeping a container alive across local test runs for speed) is a valid local-development optimization, but in CI every run starts from a known-clean environment; reuse in CI adds a class of "which run left this state behind" bugs without any real speed benefit, since CI runners are already fresh per job.

### 5. Publish and Verify Consumer-Driven Contracts in CI, Not Ad Hoc

**Rationale**: Contract tests only prevent breakage if they run automatically, on every change, on both the consumer's and the provider's side - an occasionally-run manual contract check is a false sense of safety.

```python
# consumer side: pact-python records the expected interaction
from pact import Consumer, Provider

pact = Consumer("order-service").has_pact_with(Provider("catalog-service"))

def test_get_product_contract():
    (pact
        .given("product 42 exists")
        .upon_receiving("a request for product 42")
        .with_request("GET", "/products/42")
        .will_respond_with(200, body={"id": 42, "name": "Widget", "price": "9.99"}))

    with pact:
        response = catalog_client.get_product(42)
        assert response.name == "Widget"
```

```
# provider side (catalog-service CI): verify it satisfies every published contract
pact-verifier --provider-base-url http://localhost:8000 \
              --pact-broker-base-url https://pact-broker.internal \
              --provider-app-version $GIT_SHA \
              --publish-verification-results
```

Gate deployment with the broker's `can-i-deploy` check so a provider cannot ship a contract-breaking change, and a consumer cannot deploy against a provider version it was never verified against.

### 6. Keep Integration Tests in a Separate, Explicitly Slower CI Stage

**Rationale**: Mixing fast unit tests and slower, Docker-dependent integration tests in one `pytest` invocation makes the fast feedback loop slow for everyone. Mark and separate them so the unit suite remains a sub-second local sanity check.

```python
# pytest.ini
[pytest]
markers =
    integration: real-dependency tests requiring Docker
```

```
pytest -m "not integration"   # fast local loop
pytest -m integration          # dedicated CI stage
```

## Common Pitfalls & Anti-Patterns

### Pitfall: Testing Against a Shared, Long-Lived "Test Database"

**Problem**: A shared staging database accumulates state across test runs from multiple branches/developers, causing intermittent, hard-to-reproduce failures that have nothing to do with the code under test.

**Recommended approach**: Use ephemeral, per-run containers (Testcontainers) so every run starts from an identical, known-empty state.

### Pitfall: Mocking the Boundary You Are Supposed to Be Testing

**Problem**: An "integration test" for a repository that mocks the database client entirely tests nothing more than the unit test already did - it verifies the mock was called with the right SQL string, not that the SQL is correct against a real engine.

**Recommended approach**: The entire point of an integration test is to use the real dependency (or the closest faithful substitute, e.g., a real Postgres, not SQLite, if the production target is Postgres and uses Postgres-specific features).

### Pitfall: Broad End-to-End Tests Used as the Primary Safety Net Between Services

**Problem**: Standing up every microservice to test one user journey is slow, flaky under partial infrastructure failure, and any failure could be caused by any of a dozen services - offering poor localization of the actual defect.

**Recommended approach**: Replace most cross-service assertions with contract tests (fast, localized, run independently per service) and reserve full end-to-end tests for a handful of the most business-critical journeys. See [Microservices](../architecture/microservices.md#testing-strategies).

### Pitfall: `sleep()`-Based Waiting for Container Readiness

**Problem**: Flaky under CI load (too short) or wastes time everywhere else (too long, "just in case").

**Recommended approach**: Use the library's built-in wait strategy (log pattern, HTTP health check, port open) as shown above.

### Pitfall: Skipping Contract Verification on the Provider Side

**Problem**: Teams sometimes write consumer-side contract tests but never wire the provider to verify against the broker, so a provider change can silently break every consumer's assumption with no CI signal.

**Recommended approach**: Contract testing only works as a two-sided discipline; make provider verification a required CI check, not optional tooling.

## Testing Strategies

- **Layer integration tests by boundary type**: database-adapter tests, HTTP-client tests (using a local mock server or `respx`/`aioresponses` for third-party APIs you do not own the infrastructure for), and message-broker tests (real Kafka/RabbitMQ via Testcontainers) are each their own suite with their own fixtures.
- **Reuse the same in-memory fake used for unit tests as a "contract reference"**: run one shared behavioral test suite against both the fake (fast, in unit tests) and the real adapter (in integration tests) to guarantee they never drift - see [Mocking Strategies](mocking_strategies.md#fakes-vs-mocks-in-practice).
- **Contract tests belong in both the consumer's and the provider's own CI**, never in a separate, third, cross-team pipeline that nobody owns.
- **Database migration tests**: Run schema migrations against a fresh ephemeral database as part of the integration suite to catch a broken migration before it reaches a real environment.

**Recommended test stack (Python)**: `pytest` with a dedicated `integration` marker, `testcontainers-python` for Postgres/Kafka/Redis, `respx`/`aioresponses` for third-party HTTP dependencies you cannot containerize, `pact-python` for consumer-driven contracts.

## Performance, Security & Scalability Considerations

- **CI runtime budget**: Integration test stages typically run in a separate CI job/stage from unit tests, in parallel where infrastructure allows, so the overall pipeline latency stays acceptable - see [CI/CD Patterns](../devops/ci_cd_patterns.md).
- **Resource limits in CI**: Containers spun up per test run consume real CPU/memory on CI runners; size the runner appropriately and clean up containers aggressively (context managers, not manual teardown) to avoid resource exhaustion across a busy CI fleet.
- **Secrets in test infrastructure**: Ephemeral containers should use throwaway, clearly-fake credentials - never reuse real production secrets in a test container, even a disposable one.
- **Contract testing reduces production incident risk directly**: Because it catches breaking changes before deployment rather than after, it is one of the highest-leverage practices for reducing cross-service outages in a microservices estate.

## Edge Cases & Advanced Usage

- **Testing eventual consistency**: For an asynchronous boundary (a message consumer, an outbox relay), assert the eventual state with a bounded polling/retry loop (`wait_until(condition, timeout=5)`) rather than a fixed sleep, since exact timing is not guaranteed.
- **Testing the transactional outbox pattern**: Verify, in one integration test, that a business-transaction commit and the corresponding outbox row appear atomically, and in a second test that the relay process correctly publishes and marks rows processed - see [Microservices](../architecture/microservices.md#3-use-the-transactional-outbox-pattern-to-publish-events-reliably).
- **Multi-container scenarios**: Testcontainers supports Docker Compose-based multi-container setups for tests that genuinely need more than one real dependency simultaneously (e.g., a service plus its message broker) - use sparingly, since this reintroduces some of the end-to-end trade-off.
- **Testing against cloud emulators**: For managed cloud services without a natural container (e.g., some queue or storage services), use the vendor's official emulator (LocalStack for AWS, the Firestore emulator, etc.) under the same ephemeral-container discipline.
- **Schema/contract evolution testing**: Contract tests should include a compatibility check for additive vs. breaking changes, ideally enforced by a schema registry with configured compatibility mode (backward, forward, or full).

## When *Not* to Use / Alternatives

| Dimension                              | Integration Tests                 | Unit Tests (with fakes)      | Full End-to-End Tests        |
|-------------------------------------------|--------------------------------------|----------------------------------|---------------------------------|
| Verifies real dependency behavior          | Yes                                  | No (by design)                   | Yes |
| Speed                                       | Seconds (container startup)          | Milliseconds                     | Seconds to minutes |
| Failure localization                        | Good (one boundary)                  | Excellent                        | Poor (many moving parts) |
| Infrastructure dependency (Docker, CI runners) | Required                          | None                              | Extensive |
| Appropriate volume                          | Moderate (middle of the pyramid)     | Majority                          | Very few |

**Prefer pure unit tests with fakes** when the logic under test does not actually depend on the real behavior of the external system - most business-rule logic falls here and should stay in the unit layer for speed (see [Unit Testing Principles](unit_testing_principles.md#the-test-pyramid)).

**Prefer contract tests over full end-to-end tests** for verifying cross-service compatibility - they give the same breaking-change protection at a fraction of the cost and flakiness.

## Related Documentation

- [Unit Testing Principles](unit_testing_principles.md) - the base of the pyramid this document builds on.
- [Mocking Strategies](mocking_strategies.md) - choosing fakes vs. mocks for the parts of a test that remain isolated.
- [Microservices Architecture](../architecture/microservices.md) - service boundaries this testing strategy is designed to validate.
- [Hexagonal Architecture](../architecture/hexagonal.md) - ports/adapters define exactly which boundary an integration test targets.
- [CI/CD Patterns](../devops/ci_cd_patterns.md) - pipeline staging for fast unit vs. slower integration/contract stages.
- [FastAPI Best Practices](../../languages/python/frameworks/fastapi/best_practices.md#testing-strategies)

## Glossary

- **Boundary**: A point where code interacts with a system it does not fully control (database, external API, message broker, filesystem, clock).
- **Ephemeral Test Infrastructure**: Real, disposable instances of a dependency started for the duration of a test run and destroyed afterward.
- **Testcontainers**: A family of libraries for programmatically managing ephemeral Docker containers from test code, with built-in readiness/wait strategies.
- **Contract Testing**: Verifying that a consumer's expectations of a provider's API/event shape match the provider's actual behavior, without running both systems together.
- **Consumer-Driven Contract**: A contract authored from the consumer's actual usage and published to a broker for the provider to verify against.
- **Can-I-Deploy Check**: A Pact Broker query that verifies a given service version is compatible with all currently deployed versions of its counterparts before allowing deployment.