--- title: "CI/CD Pipeline Design Patterns and Progressive Delivery" description: "Pipeline security misconfigurations documented in real 2025-2026 breach postmortems - tj-actions/changed-files, Pwn Request pull_request_target attacks, GitHub Actions cache poisoning, npm worm supply-chain incidents - plus a condensed reference for stage design, branching, and progressive delivery." language: null framework: null category: ci_cd tags: - ci-cd - devops - deployment - supply-chain-security - github-actions - trunk-based-development - canary-deployment - blue-green-deployment - feature-flags keywords: - tj-actions changed-files cve-2025-30066 - github actions pwn request pull_request_target - github actions cache poisoning oidc token theft - npm supply chain worm shai-hulud - pin github actions to commit sha - canary deployment vs blue-green deployment - trunk-based development vs gitflow last_updated: "2026-07-08" difficulty: intermediate related: - ../principles/solid.md - ../../languages/python/frameworks/fastapi/best_practices.md - ./docker_best_practices.md - ./autonomous_least_privilege_environments.md - ../security/secure_coding.md - ../security/owasp_top_10.md - ../../tools/git/best_practices.md - ../../tools/docker/dockerfile_best_practices.md - ../architecture/microservices.md - ../testing/integration_testing.md search_priority: high status: published --- # CI/CD Pipeline Design Patterns and Progressive Delivery Assumes familiarity with pipeline-as-code, trunk-based development, and canary/blue-green deployment as concepts. This document is weighted toward pipeline **security failure modes documented in real 2025-2026 incidents**, since that is where generic CI/CD knowledge is least reliable and misconfiguration cost is highest. ## Pipeline security incidents that changed default advice | Incident | Date | Mechanism | Root defect | |---|---|---|---| | **tj-actions/changed-files** - CVE-2025-30066 (GHSA-mrrh-fwg8-r2c3) | Mar 14-15, 2025 | Attacker compromised a maintainer bot's GitHub PAT, retroactively rewrote existing **version tags** (not just HEAD) to point at a malicious commit that dumped CI runner process memory (secrets, PATs, npm tokens, private RSA keys) into public build logs | Action was referenced by mutable tag (`@v45`), not immutable commit SHA - 23,000+ repositories affected | | **reviewdog/action-setup@v1** - CVE-2025-30154 | Mar 2025, disclosed alongside the above | A separate compromised action, used as the initial foothold that enabled the tj-actions compromise | Same class of defect: tag-pinned third-party action | | **pgai** - GHSA-89qq-hgvp-x37m | Mar 21 - May 14, 2025 | `pull_request_target` workflow checked out and ran untrusted PR code with base-repo secrets in scope ("Pwn Request") - full workflow secret set including a write-scoped `GITHUB_TOKEN` was exfiltratable | Classic Pwn Request: `pull_request_target` + PR-head checkout + secrets, in the same job | | **Spotipy** - CVE-2025-47928 (critical) | 2025 | Same Pwn Request pattern; researchers achieved full repository takeover | Same as above | | **TanStack npm packages** - CVE-2026-45321 / GHSA-g7cv-rxg3-hmpx (critical), attributed to "TeamPCP" | May 11, 2026 | Chained a Pwn Request into **GitHub Actions cache poisoning across the fork↔base trust boundary**, then extracted a live OIDC token from runner process memory at runtime; 84 artifacts across 42 `@tanstack/*` packages compromised. First documented malicious package shipped with **valid SLSA provenance** | Cache poisoning crossed a trust boundary that provenance/SBOM tooling does not model - proves attestation is not a substitute for pipeline isolation | | **Cline ("Clinejection")** | Dec 21, 2025 - Feb 17, 2026 | A single malicious **GitHub issue title** triggered prompt injection against Cline's AI issue-triage workflow, which pivoted via cache poisoning into the release workflows and stole `VSCE_PAT`, `OVSX_PAT`, `NPM_RELEASE_TOKEN`; a tampered Cline CLI npm package (bundling an unauthorized second agent, `openclaw`, via `postinstall`) reached ~4,000 developer machines before deprecation | An LLM-driven CI bot consuming untrusted external text (issue titles) had standing access to release-signing secrets in the same trust boundary | Direct precedent for any CI step that feeds untrusted content to an LLM - see [Autonomous, Least-Privilege, Portable Execution Environments](./autonomous_least_privilege_environments.md) | | **Shai-Hulud npm worm** | Sep 2025 (initial), recurrence through Nov 2025 | Self-replicating malware: compromised packages published further-compromised versions of their own dependents automatically | Ecosystem-scale automated propagation, not a single-target breach | | **axios** | Mar 31, 2026 | Compromised maintainer account published 2 malicious versions injecting a `plain-crypto-js` dependency that dropped a cross-platform RAT | Account takeover, not a code vulnerability - 2FA/publish-token hygiene is the control, not code review | **GitHub's structural fix**: announced Nov 7, 2025, effective Dec 8, 2025 - `pull_request_target` workflows lost implicit access to the base repo's secrets/write token when triggered from a fork PR unless explicitly reconfigured, closing the default Pwn Request shape. Pipelines written before this date should be re-audited even if "nothing changed" in their own YAML. ## Hardening checklist, ranked by what the incidents above actually exploited 1. **Pin every third-party Action to a full commit SHA, never a tag or branch.** Tags are mutable and were the exact vector in tj-actions (CVE-2025-30066) - `uses: org/action@a1b2c3d...`, with a pinned version as a comment for humans, not as the ref. 2. **Never combine `pull_request_target` (or any privileged trigger) with checkout of PR head content in the same job.** If untrusted code must run, do it under plain `pull_request` (no secrets, read-only `GITHUB_TOKEN`) and gate any privileged follow-up behind a separate, reviewed workflow (`workflow_run`) that does not execute PR-controlled code. 3. **Set minimal `permissions:` at the workflow level (`contents: read` or `{}`), escalate per-job only where needed.** Default `GITHUB_TOKEN` scope should never be write-by-default for a workflow that touches untrusted input. 4. **Scope and isolate Actions cache keys across trust boundaries.** The TanStack and Cline incidents both pivoted via cache poisoning - a cache entry written by a fork-PR job must not be readable by a job that later runs with base-repo secrets. 5. **Use OIDC federation for cloud/registry credentials, not standing PATs or static keys** - short-lived, scoped, and unusable once the job ends, unlike the bot PAT compromised in the tj-actions incident. 6. **Any CI step that feeds untrusted external text (issue titles, PR descriptions, commit messages) to an LLM must run with zero standing secrets in its execution context** - this is the Clinejection lesson specifically, and applies to any AI-agent-driven triage/review bot, not just Cline's. 7. **Treat attestation/SBOM/provenance as detection, not prevention.** TanStack shipped with valid SLSA provenance; it did not stop the compromise. Provenance answers "what did the pipeline claim to produce," not "was the pipeline itself compromised mid-run." ## Condensed reference: branching and progressive delivery | Dimension | Trunk-Based Development | GitHub Flow | GitFlow | |---|---|---|---| | Long-lived branches | None | Short-lived feature branches | `develop`/`release/*`/`hotfix/*` - long-lived | | Release cadence | Continuous | Continuous/on-demand | Scheduled, batched | | Feature flag dependency | Required for incomplete work | Common | Rarely used | | Best fit | High-velocity continuous deployment | SaaS/web apps | Versioned release trains (firmware, regulated) | | Deployment method | Rollback mechanism | Typical rollback time | |---|---|---| | Rolling update | `kubectl rollout undo` | Seconds-minutes | | Canary | Weight → 0% | Seconds | | Blue-green | Switch router back | Seconds | | Feature flag | Disable flag | Instant | | Database migration | Pre-written down-migration or expand/contract | Minutes-hours - usually the real bottleneck | Canary weight progression example (Argo Rollouts/Flagger style): `5% → pause 10m + automated error-rate/latency analysis → 25% → pause 10m → 100%`, with automated rollback on threshold breach rather than relying on a human noticing a dashboard. Database schema changes under progressive delivery: use **expand/contract** - add nullable columns/new tables, backfill and dual-write, cut reads over, remove old structures in a later deploy - so both old and new app versions run against the same schema mid-rollout. ## When *Not* to Use Continuous Deployment | Dimension | Continuous Deployment + Progressive Delivery | Scheduled/Manual Release Gate | |---|---|---| | Required for | Fast-moving SaaS, high-trust automated test suites | Regulated software, embedded/firmware, contractual release windows | | Blast radius on regression | Small (canary limits exposure) | Can be large if manual testing missed an issue | Pipeline-as-code and staged CI still apply under a manual gate - only the final deployment trigger becomes a reviewed, human-approved action instead of automatic. ## Related Documentation - [Docker Best Practices](./docker_best_practices.md) - image build artifacts and signing this pipeline produces - [Autonomous, Least-Privilege, Portable Execution Environments](./autonomous_least_privilege_environments.md) - ephemeral credentials and agent-driven CI steps, directly relevant to the Clinejection case above - [Secure Coding Practices](../security/secure_coding.md) - pipeline credential and secrets handling - [OWASP Top 10](../security/owasp_top_10.md) - software supply chain failures category ## Sources - [CISA: Supply Chain Compromise of tj-actions/changed-files (CVE-2025-30066) and reviewdog/action-setup@v1 (CVE-2025-30154)](https://www.cisa.gov/news-events/alerts/2025/03/18/supply-chain-compromise-third-party-tj-actionschanged-files-cve-2025-30066-and-reviewdogaction) - [Wiz: tj-actions/changed-files supply chain attack](https://www.wiz.io/blog/github-action-tj-actions-changed-files-supply-chain-attack-cve-2025-30066) - [GitHub Security Lab: Preventing Pwn Requests](https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/) - [GitHub Docs: Securely using pull_request_target](https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target) - [pgai security advisory GHSA-89qq-hgvp-x37m](https://github.com/timescale/pgai/security/advisories/GHSA-89qq-hgvp-x37m) - [Rescana: TanStack npm Supply Chain Attack analysis (May 2026)](https://www.rescana.com/post/tanstack-npm-supply-chain-attack-detailed-analysis-of-the-may-2026-github-actions-breach-and-multi-ecosystem-impact) - [TanStack official postmortem](https://tanstack.com/blog/npm-supply-chain-compromise-postmortem) - [Snyk: TanStack npm Packages Hit by Mini Shai-Hulud](https://snyk.io/blog/tanstack-npm-packages-compromised/) - [Adnan Khan: Clinejection](https://adnanthekhan.com/posts/clinejection/) - [Snyk: How "Clinejection" Turned an AI Bot into a Supply Chain Attack](https://snyk.io/blog/cline-supply-chain-attack-prompt-injection-github-actions/) - [Unit 42: "Shai-Hulud" Worm Compromises npm Ecosystem](https://unit42.paloaltonetworks.com/npm-supply-chain-attack/) - [Huntress: axios npm Supply Chain Compromise](https://www.huntress.com/blog/axios-npm-compromise)