core/devops/ci_cd_patterns.md

CI/CD Pipeline Design Patterns and Progressive Delivery

Pipeline security misconfigurations documented in real 2025-2026 breach postmortems - tj-actions/changed-files, Pwn Request pull_request_target attacks, GitHub Actions cache poisoning, npm worm supply-chain incidents - plus a condensed reference for stage design, branching, and progressive delivery.

ci-cddevopsdeploymentsupply-chain-securitygithub-actionstrunk-based-developmentcanary-deploymentblue-green-deploymentfeature-flags

Assumes familiarity with pipeline-as-code, trunk-based development, and canary/blue-green deployment as concepts. This document is weighted toward pipeline security failure modes documented in real 2025-2026 incidents, since that is where generic CI/CD knowledge is least reliable and misconfiguration cost is highest.

Pipeline security incidents that changed default advice

IncidentDateMechanismRoot defect
tj-actions/changed-files - CVE-2025-30066 (GHSA-mrrh-fwg8-r2c3)Mar 14-15, 2025Attacker compromised a maintainer bot's GitHub PAT, retroactively rewrote existing version tags (not just HEAD) to point at a malicious commit that dumped CI runner process memory (secrets, PATs, npm tokens, private RSA keys) into public build logsAction was referenced by mutable tag (@v45), not immutable commit SHA - 23,000+ repositories affected
reviewdog/action-setup@v1 - CVE-2025-30154Mar 2025, disclosed alongside the aboveA separate compromised action, used as the initial foothold that enabled the tj-actions compromiseSame class of defect: tag-pinned third-party action
pgai - GHSA-89qq-hgvp-x37mMar 21 - May 14, 2025pullrequesttarget workflow checked out and ran untrusted PR code with base-repo secrets in scope ("Pwn Request") - full workflow secret set including a write-scoped GITHUB_TOKEN was exfiltratableClassic Pwn Request: pullrequesttarget + PR-head checkout + secrets, in the same job
Spotipy - CVE-2025-47928 (critical)2025Same Pwn Request pattern; researchers achieved full repository takeoverSame as above
TanStack npm packages - CVE-2026-45321 / GHSA-g7cv-rxg3-hmpx (critical), attributed to "TeamPCP"May 11, 2026Chained a Pwn Request into GitHub Actions cache poisoning across the fork↔base trust boundary, then extracted a live OIDC token from runner process memory at runtime; 84 artifacts across 42 @tanstack/* packages compromised. First documented malicious package shipped with valid SLSA provenanceCache poisoning crossed a trust boundary that provenance/SBOM tooling does not model - proves attestation is not a substitute for pipeline isolation
Cline ("Clinejection")Dec 21, 2025 - Feb 17, 2026A single malicious GitHub issue title triggered prompt injection against Cline's AI issue-triage workflow, which pivoted via cache poisoning into the release workflows and stole VSCEPAT, OVSXPAT, NPMRELEASETOKEN; a tampered Cline CLI npm package (bundling an unauthorized second agent, openclaw, via postinstall) reached ~4,000 developer machines before deprecationAn LLM-driven CI bot consuming untrusted external text (issue titles) had standing access to release-signing secrets in the same trust boundaryDirect precedent for any CI step that feeds untrusted content to an LLM - see Autonomous, Least-Privilege, Portable Execution Environments
Shai-Hulud npm wormSep 2025 (initial), recurrence through Nov 2025Self-replicating malware: compromised packages published further-compromised versions of their own dependents automaticallyEcosystem-scale automated propagation, not a single-target breach
axiosMar 31, 2026Compromised maintainer account published 2 malicious versions injecting a plain-crypto-js dependency that dropped a cross-platform RATAccount takeover, not a code vulnerability - 2FA/publish-token hygiene is the control, not code review

GitHub's structural fix: announced Nov 7, 2025, effective Dec 8, 2025 - pullrequesttarget workflows lost implicit access to the base repo's secrets/write token when triggered from a fork PR unless explicitly reconfigured, closing the default Pwn Request shape. Pipelines written before this date should be re-audited even if "nothing changed" in their own YAML.

Hardening checklist, ranked by what the incidents above actually exploited

  1. Pin every third-party Action to a full commit SHA, never a tag or branch. Tags are mutable and were the exact vector in tj-actions (CVE-2025-30066) - uses: org/action@a1b2c3d..., with a pinned version as a comment for humans, not as the ref.
  2. Never combine pullrequesttarget (or any privileged trigger) with checkout of PR head content in the same job. If untrusted code must run, do it under plain pullrequest (no secrets, read-only GITHUBTOKEN) and gate any privileged follow-up behind a separate, reviewed workflow (workflow_run) that does not execute PR-controlled code.
  3. Set minimal permissions: at the workflow level (contents: read or {}), escalate per-job only where needed. Default GITHUB_TOKEN scope should never be write-by-default for a workflow that touches untrusted input.
  4. Scope and isolate Actions cache keys across trust boundaries. The TanStack and Cline incidents both pivoted via cache poisoning - a cache entry written by a fork-PR job must not be readable by a job that later runs with base-repo secrets.
  5. Use OIDC federation for cloud/registry credentials, not standing PATs or static keys - short-lived, scoped, and unusable once the job ends, unlike the bot PAT compromised in the tj-actions incident.
  6. Any CI step that feeds untrusted external text (issue titles, PR descriptions, commit messages) to an LLM must run with zero standing secrets in its execution context - this is the Clinejection lesson specifically, and applies to any AI-agent-driven triage/review bot, not just Cline's.
  7. Treat attestation/SBOM/provenance as detection, not prevention. TanStack shipped with valid SLSA provenance; it did not stop the compromise. Provenance answers "what did the pipeline claim to produce," not "was the pipeline itself compromised mid-run."

Condensed reference: branching and progressive delivery

DimensionTrunk-Based DevelopmentGitHub FlowGitFlow
Long-lived branchesNoneShort-lived feature branchesdevelop/release//hotfix/ - long-lived
Release cadenceContinuousContinuous/on-demandScheduled, batched
Feature flag dependencyRequired for incomplete workCommonRarely used
Best fitHigh-velocity continuous deploymentSaaS/web appsVersioned release trains (firmware, regulated)
Deployment methodRollback mechanismTypical rollback time
Rolling updatekubectl rollout undoSeconds-minutes
CanaryWeight → 0%Seconds
Blue-greenSwitch router backSeconds
Feature flagDisable flagInstant
Database migrationPre-written down-migration or expand/contractMinutes-hours - usually the real bottleneck

Canary weight progression example (Argo Rollouts/Flagger style): 5% → pause 10m + automated error-rate/latency analysis → 25% → pause 10m → 100%, with automated rollback on threshold breach rather than relying on a human noticing a dashboard.

Database schema changes under progressive delivery: use expand/contract - add nullable columns/new tables, backfill and dual-write, cut reads over, remove old structures in a later deploy - so both old and new app versions run against the same schema mid-rollout.

When Not to Use Continuous Deployment

DimensionContinuous Deployment + Progressive DeliveryScheduled/Manual Release Gate
Required forFast-moving SaaS, high-trust automated test suitesRegulated software, embedded/firmware, contractual release windows
Blast radius on regressionSmall (canary limits exposure)Can be large if manual testing missed an issue

Pipeline-as-code and staged CI still apply under a manual gate - only the final deployment trigger becomes a reviewed, human-approved action instead of automatic.

Sources