---
title: "Docker Best Practices for Production Deployment and CI/CD"
description: "Non-default hardening flags for Docker/OCI image builds: BuildKit SBOM/provenance attestation, cosign/Sigstore signing, rootless-mode trade-offs, distroless CVE deltas with real numbers, and named supply-chain incidents (malicious Docker Hub images, runc CVE-2024-21626)."
language: null
framework: null
category: devops_general
tags:
  - docker
  - containers
  - deployment
  - ci-cd
  - security
  - supply-chain
  - sbom
  - provenance
  - best-practices
  - kubernetes
keywords:
  - docker buildkit sbom provenance attestation flags
  - cosign sigstore keyless signing
  - distroless vs debian slim cve count
  - rootless docker seccomp apparmor trade-off
  - runc cve-2024-21626 container breakout
  - malicious docker hub images typosquatting
  - docker image scanning trivy grype
last_updated: "2026-07-08"
difficulty: intermediate
version: "Docker Engine >= 25, BuildKit >= 0.14"
related:
  - ../principles/solid.md
  - ../../languages/python/frameworks/fastapi/best_practices.md
  - ./ci_cd_patterns.md
  - ./autonomous_least_privilege_environments.md
  - ../security/secure_coding.md
  - ../security/owasp_top_10.md
  - ../../tools/git/best_practices.md
  - ../../tools/docker/dockerfile_best_practices.md
  - ../architecture/microservices.md
  - ../testing/integration_testing.md
search_priority: high
status: published
---

# Docker Best Practices for Production Deployment and CI/CD

Assumes familiarity with multi-stage builds, layer caching, and Dockerfile syntax. This document covers only the parts that are non-default, security-load-bearing, or backed by a named 2024-2026 incident. See [Dockerfile Best Practices](../../tools/docker/dockerfile_best_practices.md) for tactical instruction authoring.

## Non-default flags that actually change your risk posture

| Flag / config | Effect | Why it's not the default |
|---|---|---|
| `docker buildx build --sbom=true --provenance=mode=max` | Attaches in-toto SBOM + max-detail build provenance to the image manifest | `docker buildx build` emits **no** attestations unless both flags are passed explicitly |
| `--provenance=mode=max` vs `mode=min` | `max` records full build definition (sources, build args, env) vs `min`'s bare metadata | `max` can leak build-arg values into the attestation - never pass secrets via `--build-arg` |
| `RUN --mount=type=secret,id=token` | Secret is mounted only for that `RUN` step, never written to a layer | Default `ARG`/`ENV` values persist permanently in image history, recoverable via `docker history` or layer tarball extraction |
| `RUN --mount=type=cache,target=...` | Persistent build cache across builds without polluting the image layer | Ordinary `RUN` output always becomes part of the layer |
| `FROM image@sha256:...` (digest pin) | Immune to upstream tag repointing | Tags (`:latest`, even `:3.13-slim`) are mutable pointers, digests are not |
| `docker buildx build --platform linux/amd64,linux/arm64` | Single manifest list, one build | Default build targets only the local architecture |
| Rootless daemon (`dockerd-rootless.sh`) | Daemon itself runs as an unprivileged host user | Standard install runs `dockerd` as root; a daemon compromise is a host-root compromise |

**Docker's attestation format is not natively Sigstore/Notary-compatible.** BuildKit's in-toto attestations use Docker's own attachment convention, not the Sigstore or Notation image-signing spec - no external verification tool reads them automatically. Signing (below) is the actual chain-of-custody mechanism; attestations alone answer "what's in this image," not "was this image tampered with."

## Signing: cosign + Sigstore, correctly

```bash
docker buildx build --sbom=true --provenance=mode=max \
  -t myregistry.example.com/myapp@sha256:<digest> --push .
cosign sign myregistry.example.com/myapp@sha256:<digest>   # keyless: short-lived cert from Fulcio via CI's OIDC identity
```

- Sign **by digest, not by tag** - Sigstore is deprecating tag-based signing/attestation because a tag can be repointed after signing, invalidating the guarantee. Always resolve to `@sha256:...` before `cosign sign`.
- Keyless signing (no `--key`) ties the signature to the CI job's OIDC identity via Fulcio's short-lived certificates and a public transparency log (Rekor) - no long-lived signing key to leak, matching the ephemeral-credential principle in [Autonomous, Least-Privilege, Portable Execution Environments](./autonomous_least_privilege_environments.md).
- Enforce at admission, not just at push: Kyverno or the Sigstore Policy Controller rejecting unsigned images at the Kubernetes scheduler is what actually closes the loop - a signed-but-unenforced image provides an audit trail, not a control.

## Base image choice: the CVE delta is now measured, not assumed

| Base tier | Typical size | Measured CVE exposure |
|---|---|---|
| Debian-based Docker Hub images (general population) | Varies | ~280 known CVEs on average across a scanned sample |
| `python:3.13-slim` (Debian stable) | ~130 MB | glibc/OpenSSL layer CVEs accumulate for weeks-to-months before a Debian point release fixes them - a 50-image scan in early 2026 found the distroless-tier Python image carrying 7 HIGH-severity CVEs at scan time |
| `gcr.io/distroless/*` | 2-20 MB | Same Debian-derived package set as above - "distroless" reduces attack surface (no shell/package manager) but does **not** reduce patch lag |
| Chainguard / Wolfi-based images | Comparable or smaller (multi-layer sharing since May 2025, ~70% catalog-wide layer dedup) | Rebuilt and republished within hours of an upstream fix; the same early-2026 scan found 0 HIGH CVEs on the equivalent Chainguard Python image |

Takeaway: "distroless" and "low-CVE" are not the same property. Distroless shrinks attack surface (no shell to pivot through); a continuously-rebuilt base (Chainguard/Wolfi) is what actually shrinks the CVE window. Use both criteria independently when picking a base.

## Named incidents that should change your defaults

| Incident | What happened | Defense |
|---|---|---|
| **runc CVE-2024-21626** (container breakout, fixed in runc 1.1.12) | A leaked file descriptor to the host's `/sys/fs/cgroup` (exposed via `/proc/self/fd/`) let a container process set its working directory into the host filesystem namespace via `process.cwd`, exploitable through `runc run` (malicious image) or `runc exec` | Pin runc/containerd/Docker Engine to patched versions; this is exactly the class of host-escape vulnerability that non-root **inside** the container does not fully mitigate - the escape happens at the runtime layer, not the in-container privilege layer |
| **Malicious Docker Hub images at scale** - JFrog (Apr 2024): millions of "imageless" repos with malicious metadata; Unit42: 20M+ pulls across cryptojacking images (~85% mining Monero via XMRig); Aqua Security (2023): 1,600+ typosquatted images impersonating popular projects; follow-on research (2024): 3M+ malicious/typosquatted repos identified | Public registries are an active, ongoing attack surface, not a one-time-vetted catalog | Pin by digest, scan on every pull (not just at build time - a clean image today can be poisoned tomorrow if you re-pull a mutable tag), prefer verified-publisher/official images, mirror through a private pull-through cache you control |
| **Rootless-mode namespace trade-off** | Running BuildKit inside a rootless daemon typically requires `seccomp=unconfined` and `apparmor=unconfined`, because Docker's default seccomp profile blocks the ~44-50 syscalls (`unshare`, `mount`, `keyctl`, `perf_event_open`, `add_key`, and full namespace-creation calls) that rootless mode itself needs to function. In 2025, Qualys disclosed three separate bypass techniques against Ubuntu's AppArmor-enforced restriction on unprivileged user-namespace creation - the exact mechanism rootless mode depends on | Rootless mode moves risk from "daemon compromise = host root" to "kernel user-namespace bug = host root"; it is not a strict security upgrade. Evaluate against your actual threat model rather than adopting it by default; gVisor/Kata/Firecracker-backed runtimes are a stronger isolation boundary for genuinely untrusted workloads (see [Autonomous, Least-Privilege, Portable Execution Environments](./autonomous_least_privilege_environments.md)) |

## Hardening checklist, ranked by measured risk reduction

1. **Pin base images by digest**, not tag - closes the mutable-tag repointing vector used in most typosquatting/registry-poisoning cases above.
2. **Scan on every pull, not just at build** - `trivy image --severity HIGH,CRITICAL --exit-code 1`; a registry image can be repointed or newly-CVE'd after your last scan.
3. **Sign by digest with keyless cosign + enforce at admission** (Kyverno/Sigstore Policy Controller) - the only control in this list that stops a compromised-but-valid-looking image from running at all.
4. **Non-root `USER <numeric-uid>` + `readOnlyRootFilesystem: true` + `cap_drop: [ALL]`** - bounds blast radius of a runtime compromise; does not stop a runtime-layer escape like CVE-2024-21626, which is why (5) matters independently.
5. **Track and patch the container runtime itself** (runc/containerd/Docker Engine versions) - the image-layer hardening above is irrelevant if the runtime underneath has an active escape CVE.
6. **BuildKit secret mounts, never `ARG`/`ENV` for credentials** - the only way to guarantee a credential never lands in a recoverable layer.
7. **`--sbom=true --provenance=mode=max`** - necessary for incident response ("are we affected by CVE-X") but not a preventive control by itself; rank below signing and scanning.

## When *Not* to Use

| Dimension | Docker Containers | Serverless Functions | Static Binaries / VMs |
|---|---|---|---|
| Cold start | Low with slim/distroless images | Can be higher for large runtimes | Instant (binary) / slow (VM) |
| Fine-grained per-request billing | No | Yes | No |
| Hard kernel-level requirements (custom modules, GPU passthrough) | Poor fit | Poor fit | Required |

Choose serverless for sporadic, short-lived, event-driven workloads. Choose a static binary or VM when the container runtime itself is unneeded overhead or the workload needs kernel features containers can't cleanly expose.

## Sources

- [BuildKit attestations and caveats - AugmentedMind.de](https://www.augmentedmind.de/2025/03/16/docker-image-attestation-buildkit/)
- [Docker Build attestations reference](https://docs.docker.com/build/metadata/attestations/)
- [Signing Other Types - Sigstore docs](https://docs.sigstore.dev/cosign/signing/other_types/)
- [Chainguard 2026: 2,000+ Zero-CVE Container Images - AppSec Santa](https://appsecsanta.com/chainguard)
- [Distroless vs Chainguard vs Wolfi - Safeguard](https://safeguard.sh/resources/blog/distroless-vs-chainguard-vs-wolfi-base-images)
- [JFrog: coordinated attacks planting millions of malicious Docker Hub repositories (Apr 2024)](https://jfrog.com/blog/attacks-on-docker-with-millions-of-malicious-repositories-spread-malware-and-phishing-scams/)
- [20 Million Miners: Malicious Cryptojacking Images in Docker Hub - Unit 42](https://unit42.paloaltonetworks.com/malicious-cryptojacking-images/)
- [Typosquatted Alpine - CleanStart](https://www.cleanstart.com/blogs/typosquatted-alpine-a-devops-cautionary-tale)
- [CVE-2024-21626 - runc working directory breakout - WithSecure Labs](https://labs.withsecure.com/publications/runc-working-directory-breakout--cve-2024-21626)
- [CVE-2024-21626 - Palo Alto Networks "Leaky Vessels"](https://www.paloaltonetworks.com/blog/cloud-security/leaky-vessels-vulnerabilities-container-escape/)
- [Rootless Docker and Its Hidden Security Trade-Offs - Ken Muse](https://www.kenmuse.com/blog/rootless-docker-and-its-hidden-security-trade-offs/)
- [Docker Security Cheat Sheet - OWASP](https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html)