core/devops/docker_best_practices.md

Docker Best Practices for Production Deployment and CI/CD

Non-default hardening flags for Docker/OCI image builds: BuildKit SBOM/provenance attestation, cosign/Sigstore signing, rootless-mode trade-offs, distroless CVE deltas with real numbers, and named supply-chain incidents (malicious Docker Hub images, runc CVE-2024-21626).

Assumes familiarity with multi-stage builds, layer caching, and Dockerfile syntax. This document covers only the parts that are non-default, security-load-bearing, or backed by a named 2024-2026 incident. See Dockerfile Best Practices for tactical instruction authoring.

Non-default flags that actually change your risk posture

Flag / config	Effect	Why it's not the default
`docker buildx build --sbom=true --provenance=mode=max`	Attaches in-toto SBOM + max-detail build provenance to the image manifest	`docker buildx build` emits no attestations unless both flags are passed explicitly
`--provenance=mode=max` vs `mode=min`	`max` records full build definition (sources, build args, env) vs `min`'s bare metadata	`max` can leak build-arg values into the attestation - never pass secrets via `--build-arg`
`RUN --mount=type=secret,id=token`	Secret is mounted only for that `RUN` step, never written to a layer	Default `ARG`/`ENV` values persist permanently in image history, recoverable via `docker history` or layer tarball extraction
`RUN --mount=type=cache,target=...`	Persistent build cache across builds without polluting the image layer	Ordinary `RUN` output always becomes part of the layer
`FROM image@sha256:...` (digest pin)	Immune to upstream tag repointing	Tags (`:latest`, even `:3.13-slim`) are mutable pointers, digests are not
`docker buildx build --platform linux/amd64,linux/arm64`	Single manifest list, one build	Default build targets only the local architecture
Rootless daemon (`dockerd-rootless.sh`)	Daemon itself runs as an unprivileged host user	Standard install runs `dockerd` as root; a daemon compromise is a host-root compromise

Docker's attestation format is not natively Sigstore/Notary-compatible. BuildKit's in-toto attestations use Docker's own attachment convention, not the Sigstore or Notation image-signing spec - no external verification tool reads them automatically. Signing (below) is the actual chain-of-custody mechanism; attestations alone answer "what's in this image," not "was this image tampered with."

Signing: cosign + Sigstore, correctly

docker buildx build --sbom=true --provenance=mode=max \
  -t myregistry.example.com/myapp@sha256:<digest> --push .
cosign sign myregistry.example.com/myapp@sha256:<digest>   # keyless: short-lived cert from Fulcio via CI's OIDC identity

Sign by digest, not by tag - Sigstore is deprecating tag-based signing/attestation because a tag can be repointed after signing, invalidating the guarantee. Always resolve to @sha256:... before cosign sign.
Keyless signing (no --key) ties the signature to the CI job's OIDC identity via Fulcio's short-lived certificates and a public transparency log (Rekor) - no long-lived signing key to leak, matching the ephemeral-credential principle in Autonomous, Least-Privilege, Portable Execution Environments.
Enforce at admission, not just at push: Kyverno or the Sigstore Policy Controller rejecting unsigned images at the Kubernetes scheduler is what actually closes the loop - a signed-but-unenforced image provides an audit trail, not a control.

Base image choice: the CVE delta is now measured, not assumed

Base tier	Typical size	Measured CVE exposure
Debian-based Docker Hub images (general population)	Varies	~280 known CVEs on average across a scanned sample
`python:3.13-slim` (Debian stable)	~130 MB	glibc/OpenSSL layer CVEs accumulate for weeks-to-months before a Debian point release fixes them - a 50-image scan in early 2026 found the distroless-tier Python image carrying 7 HIGH-severity CVEs at scan time
`gcr.io/distroless/*`	2-20 MB	Same Debian-derived package set as above - "distroless" reduces attack surface (no shell/package manager) but does not reduce patch lag
Chainguard / Wolfi-based images	Comparable or smaller (multi-layer sharing since May 2025, ~70% catalog-wide layer dedup)	Rebuilt and republished within hours of an upstream fix; the same early-2026 scan found 0 HIGH CVEs on the equivalent Chainguard Python image

Takeaway: "distroless" and "low-CVE" are not the same property. Distroless shrinks attack surface (no shell to pivot through); a continuously-rebuilt base (Chainguard/Wolfi) is what actually shrinks the CVE window. Use both criteria independently when picking a base.

Named incidents that should change your defaults

Incident	What happened	Defense
runc CVE-2024-21626 (container breakout, fixed in runc 1.1.12)	A leaked file descriptor to the host's `/sys/fs/cgroup` (exposed via `/proc/self/fd/`) let a container process set its working directory into the host filesystem namespace via `process.cwd`, exploitable through `runc run` (malicious image) or `runc exec`	Pin runc/containerd/Docker Engine to patched versions; this is exactly the class of host-escape vulnerability that non-root inside the container does not fully mitigate - the escape happens at the runtime layer, not the in-container privilege layer
Malicious Docker Hub images at scale - JFrog (Apr 2024): millions of "imageless" repos with malicious metadata; Unit42: 20M+ pulls across cryptojacking images (~85% mining Monero via XMRig); Aqua Security (2023): 1,600+ typosquatted images impersonating popular projects; follow-on research (2024): 3M+ malicious/typosquatted repos identified	Public registries are an active, ongoing attack surface, not a one-time-vetted catalog	Pin by digest, scan on every pull (not just at build time - a clean image today can be poisoned tomorrow if you re-pull a mutable tag), prefer verified-publisher/official images, mirror through a private pull-through cache you control
Rootless-mode namespace trade-off	Running BuildKit inside a rootless daemon typically requires `seccomp=unconfined` and `apparmor=unconfined`, because Docker's default seccomp profile blocks the ~44-50 syscalls (`unshare`, `mount`, `keyctl`, `perfeventopen`, `add_key`, and full namespace-creation calls) that rootless mode itself needs to function. In 2025, Qualys disclosed three separate bypass techniques against Ubuntu's AppArmor-enforced restriction on unprivileged user-namespace creation - the exact mechanism rootless mode depends on	Rootless mode moves risk from "daemon compromise = host root" to "kernel user-namespace bug = host root"; it is not a strict security upgrade. Evaluate against your actual threat model rather than adopting it by default; gVisor/Kata/Firecracker-backed runtimes are a stronger isolation boundary for genuinely untrusted workloads (see Autonomous, Least-Privilege, Portable Execution Environments)

Hardening checklist, ranked by measured risk reduction

Pin base images by digest, not tag - closes the mutable-tag repointing vector used in most typosquatting/registry-poisoning cases above.
Scan on every pull, not just at build - trivy image --severity HIGH,CRITICAL --exit-code 1; a registry image can be repointed or newly-CVE'd after your last scan.
Sign by digest with keyless cosign + enforce at admission (Kyverno/Sigstore Policy Controller) - the only control in this list that stops a compromised-but-valid-looking image from running at all.
Non-root USER <numeric-uid> + readOnlyRootFilesystem: true + cap_drop: [ALL] - bounds blast radius of a runtime compromise; does not stop a runtime-layer escape like CVE-2024-21626, which is why (5) matters independently.
Track and patch the container runtime itself (runc/containerd/Docker Engine versions) - the image-layer hardening above is irrelevant if the runtime underneath has an active escape CVE.
BuildKit secret mounts, never ARG/ENV for credentials - the only way to guarantee a credential never lands in a recoverable layer.
--sbom=true --provenance=mode=max - necessary for incident response ("are we affected by CVE-X") but not a preventive control by itself; rank below signing and scanning.

When Not to Use

Dimension	Docker Containers	Serverless Functions	Static Binaries / VMs
Cold start	Low with slim/distroless images	Can be higher for large runtimes	Instant (binary) / slow (VM)
Fine-grained per-request billing	No	Yes	No
Hard kernel-level requirements (custom modules, GPU passthrough)	Poor fit	Poor fit	Required

Choose serverless for sporadic, short-lived, event-driven workloads. Choose a static binary or VM when the container runtime itself is unneeded overhead or the workload needs kernel features containers can't cleanly expose.