-
Notifications
You must be signed in to change notification settings - Fork 1
docs: add Terraform drift detection tutorial #300
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 2 commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
29e14ed
docs: add Terraform drift detection tutorial
mintlify[bot] fb8f65e
docs: split drift detection into DIY and Kosli tutorials, remove outd…
mintlify[bot] a3743c5
docs: restructure drift detection into one tutorial per drift type
gsavage 825d08c
docs: link drift detection tutorials to SDLC-CTRL-0018
gsavage ac91e5b
docs: add name/on block to apply workflow example
gsavage 2af8ba4
docs: remove race condition notes from drift detection hardening
gsavage f130222
chore: remove orphaned drift detection images
gsavage 7a92f41
Merge branch 'main' into mintlify/9dc5c128
gsavage File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,200 @@ | ||
| --- | ||
| title: Detecting Terraform drift (do-it-yourself) | ||
| description: Build a minimal, vendor-neutral Terraform drift detector — the CI-agnostic loop that catches world changes with a scheduled plan against the applied SHA. | ||
| --- | ||
|
|
||
| <Tooltip tip="Drift occurs when infrastructure is changed directly via cloud APIs, consoles, or CLI tools without going through Terraform, causing the actual state to diverge from the desired state defined in your Terraform config.">Configuration drift</Tooltip> is the gap between what Terraform believes is deployed and what is actually running. The moment a change is made outside your approved process — a hotfix in the cloud console, a partial apply failure, an out-of-band automation — your version-controlled baseline silently stops matching reality. | ||
|
|
||
| This tutorial walks through the minimum viable drift detector: a scheduled Terraform plan against the last-applied git SHA, with no external tooling. It ports to any CI system; the worked example uses GitHub Actions. | ||
|
|
||
| <Warning> | ||
| A bare scheduled plan catches **world changes** — a console or API edit the statefile doesn't know about. It cannot catch an **out-of-CI apply**: a laptop apply updates the statefile and the world together, so they still agree and the plan stays empty. Closing that gap needs the statefile's provenance recorded and checked continuously — see [Detecting Terraform drift with Kosli](/tutorials/terraform_drift_detection_kosli) for the two-mechanism control with audit-grade evidence. | ||
| </Warning> | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| Fix these first — drift detection on top of an undisciplined apply process produces mostly noise. | ||
|
|
||
| - Terraform is applied through CI/CD, not from laptops, as the normal path. | ||
| - Remote, locked state — for example, an S3 backend with the native S3 lockfile (or DynamoDB). | ||
| - Keyless CI authentication to your cloud (for example, GitHub OIDC) with a dedicated, read-capable role for the detector. The detector never needs apply permissions. | ||
| - A notification channel (Slack, Teams, email) and a scheduler (GitHub Actions schedule, GitLab scheduled pipelines, or cron). | ||
|
|
||
| ## Record a baseline at apply time | ||
|
|
||
| Every time the pipeline applies, record the exact git SHA that was applied, tied to the environment, somewhere durable. A small JSON marker stored next to the statefile works well — call it `drift.plan.json`: | ||
|
|
||
| ```json | ||
| { | ||
| "sha": "abc123def456...", | ||
| "drift": false | ||
| } | ||
| ``` | ||
|
|
||
| The `drift` field starts as `false` (no drift) and doubles as a latch — see [Latch, don't spam](#latch-dont-spam) below. | ||
|
|
||
| Why store a SHA at all, rather than "just plan against `main`"? Because applying and merging are not the same event — see [Plan against the applied SHA](#plan-against-the-applied-sha-not-against-main) below. | ||
|
|
||
| ## Detect divergence with a scheduled plan | ||
|
|
||
| On a schedule, run `terraform init` and `terraform plan` against the recorded baseline, then inspect the result. The cleanest machine-readable signal is the plan exit code: | ||
|
|
||
| ```shell | ||
| terraform plan -input=false -lock=false -detailed-exitcode -no-color -out=tfplan | ||
| # exit 0 -> no changes (no drift) | ||
| # exit 2 -> changes present (DRIFT) | ||
| # exit 1 -> error | ||
| ``` | ||
|
|
||
| Use `-lock=false` so the read-only drift plan never contends with a real apply, and `-input=false` so it can never hang waiting for a prompt. | ||
|
|
||
| ## Plan against the applied SHA, not against `main` | ||
|
|
||
| This is the single most common false-positive source. If changes are merged to `main` but not yet applied — because the apply is gated behind a manual approval, or batched into a release — then planning against `main` shows a non-empty plan that reflects pending intentional changes, not drift. Always check out the recorded baseline SHA before planning: | ||
|
|
||
| ```yaml | ||
| - name: Check out the applied commit | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| ref: ${{ steps.baseline.outputs.sha }} | ||
| ``` | ||
|
|
||
| ## Surface the result and alert | ||
|
|
||
| When drift is found, notify your channel and optionally record the event (for example, by stamping the marker with a timestamp). Keep the signal low-noise: drift is usually a detective control over an environment already protected by preventative controls, so it rarely warrants paging an on-call engineer. Best-effort triage by the team is normally the right posture. | ||
|
|
||
| ## A complete worked example | ||
|
|
||
| A self-contained GitHub Actions workflow with no external tooling. It reads the baseline SHA from S3, checks out that commit, plans, and alerts on a non-empty plan. | ||
|
|
||
| ```yaml | ||
| name: Drift Detection | ||
| on: | ||
| schedule: | ||
| - cron: "*/30 * * * *" # every 30 minutes; tune per environment | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| id-token: write # cloud OIDC | ||
| contents: read | ||
|
|
||
| concurrency: # never let two drift runs overlap | ||
| group: drift-${{ github.repository }} | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| detect: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Configure AWS credentials (OIDC) | ||
| uses: aws-actions/configure-aws-credentials@v6 | ||
| with: | ||
| role-to-assume: ${{ vars.DRIFT_ROLE_ARN }} | ||
| aws-region: ${{ vars.AWS_REGION }} | ||
|
|
||
| - name: Read baseline SHA | ||
| id: baseline | ||
| env: | ||
| STATE_BUCKET: ${{ vars.STATE_BUCKET }} | ||
| REPO: ${{ github.event.repository.name }} | ||
| run: | | ||
| aws s3 cp "s3://$STATE_BUCKET/terraform/$REPO/drift.plan.json" /tmp/baseline.json | ||
| echo "sha=$(jq -r .sha /tmp/baseline.json)" >> "$GITHUB_OUTPUT" | ||
| echo "drift=$(jq -r .drift /tmp/baseline.json)" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Check out the applied commit | ||
| if: steps.baseline.outputs.drift == 'false' # latch: skip if already flagged | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| ref: ${{ steps.baseline.outputs.sha }} | ||
|
|
||
| - uses: hashicorp/setup-terraform@v4 | ||
| with: | ||
| terraform_wrapper: false | ||
|
|
||
| - name: Init & plan | ||
| if: steps.baseline.outputs.drift == 'false' | ||
| id: plan | ||
| run: | | ||
| terraform init -input=false | ||
| set +e | ||
| terraform plan -input=false -lock=false -detailed-exitcode -no-color -out=tfplan | ||
| echo "exitcode=$?" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Alert + record drift | ||
| if: steps.plan.outputs.exitcode == '2' | ||
| env: | ||
| SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }} | ||
| STATE_BUCKET: ${{ vars.STATE_BUCKET }} | ||
| REPO: ${{ github.event.repository.name }} | ||
| SHA: ${{ steps.baseline.outputs.sha }} | ||
| run: | | ||
| TS=$(date -u +%Y-%m-%dT%H:%M:%SZ) | ||
| jq -n --arg sha "$SHA" --arg drift "$TS" '{sha:$sha, drift:$drift}' \ | ||
| > /tmp/drift.plan.json | ||
| aws s3 cp /tmp/drift.plan.json \ | ||
| "s3://$STATE_BUCKET/terraform/$REPO/drift.plan.json" | ||
| curl -sf -X POST "$SLACK_WEBHOOK" -H 'Content-type: application/json' \ | ||
| --data "{\"text\":\"Drift detected in $GITHUB_REPOSITORY against $SHA\"}" | ||
| ``` | ||
|
|
||
| ## Latch, don't spam | ||
|
|
||
| Once drift is flagged, you usually don't want to re-plan and re-alert every cycle until someone acts. A simple latch lives in the marker itself: store the drift state in the `drift` field, skip planning while it is not `false`, and reset it to `false` on the next successful apply. The worked example above does exactly this with its `if: ... drift == 'false'` guards. | ||
|
|
||
| ## Hardening | ||
|
|
||
| A detector that runs once and alerts once is easy. A detector you can depend on for an audit needs to handle the failure modes below. | ||
|
|
||
| <AccordionGroup> | ||
| <Accordion title="Monitor the monitor" icon="heart-pulse"> | ||
| This is the most dangerous failure mode. If the scheduled job silently stops running, no new evidence arrives to contradict the last result — so the environment looks green forever, even as drift accumulates. Treating "the dashboard is green" as proof of cleanliness, without also verifying the underlying job is running on schedule, is a misuse of the control. Add a heartbeat or alert on "job has not run in N intervals". | ||
| </Accordion> | ||
|
|
||
| <Accordion title="Terraform-managed resources only" icon="filter"> | ||
| `terraform plan` can only see resources Terraform manages. A resource created entirely outside Terraform — say, an IAM user added by hand in the console with no corresponding Terraform resource — is invisible to this control. Closing that gap is the job of an Infrastructure-as-Code coverage policy (everything in production must be defined as code in the first place); drift detection assumes that policy holds and does not substitute for it. | ||
| </Accordion> | ||
|
|
||
| <Accordion title="Race conditions are bounded, not eliminated" icon="clock"> | ||
| Two timing windows exist in theory. First, between an apply finishing and the baseline marker being written, a concurrent unauthorised change could slip in. In practice the Terraform lock prevents a concurrent apply from starting until the lock is released, which keeps this theoretical. Second, if the world changes between one drift cycle and the next, the alert may be delayed by one interval. Understand both; rely on locking and ordering to keep them harmless. | ||
|
gsavage marked this conversation as resolved.
Outdated
|
||
| </Accordion> | ||
|
|
||
| <Accordion title="Tune cadence per environment" icon="gauge"> | ||
| Worst-case detection latency is the check interval. A ten-minute check surfaces drift within ten minutes. Set the schedule from each environment's rate-of-change and blast radius rather than using one global value. | ||
| </Accordion> | ||
|
|
||
| <Accordion title="Concurrency and least privilege" icon="lock"> | ||
| Guard against overlapping runs for the same environment with a concurrency group (as in the example above). Scope the detector's cloud role tightly: it needs to read state and plan, plus write the marker file — nothing more. It must never hold apply permissions. | ||
| </Accordion> | ||
| </AccordionGroup> | ||
|
|
||
| ## Adapting to other CI systems | ||
|
|
||
| The mechanism is CI-agnostic; only the authentication and scheduling glue changes. On any platform the loop is identical: read the baseline SHA, check out that commit, run a read-only plan, branch on whether the plan is empty. | ||
|
|
||
| - **GitLab CI** — a scheduled pipeline with OIDC to your cloud, the same checkout-by-SHA and `plan -detailed-exitcode`. | ||
| - **Jenkins / cron** — a timer-triggered job doing the same steps; store the marker in object storage or any durable key-value store. | ||
|
|
||
| ## Implementation checklist | ||
|
|
||
| - [ ] Terraform is applied through CI/CD, with remote, locked state. | ||
| - [ ] Each apply records the applied git SHA to a durable, per-environment marker. | ||
| - [ ] A scheduled job plans against that SHA — not against `main` — using a read-only, lock-free plan. | ||
| - [ ] A non-empty plan raises an alert and is recorded; the result latches until the next apply resets it. | ||
| - [ ] The scheduled job itself is monitored for silent failure (heartbeat / not-run alert). | ||
| - [ ] The detector's cloud role can read and plan only — never apply. | ||
| - [ ] Cadence and concurrency are tuned per environment. | ||
|
|
||
| ## Next steps | ||
|
|
||
| The DIY loop above catches world changes but leaves two gaps: | ||
|
|
||
| - It leaves no durable, tamper-evident record — "the job said green" is an assertion, not evidence an auditor can verify. | ||
| - It cannot detect an out-of-CI apply (a laptop apply updates the statefile and the world together, so they still agree and the plan stays empty). | ||
|
|
||
| Both gaps are closed by attesting the statefile and drift marker into a Kosli Environment. See [Detecting Terraform drift with Kosli](/tutorials/terraform_drift_detection_kosli). | ||
|
|
||
| ## Further reading | ||
|
|
||
| - [Terraform `-detailed-exitcode`](https://developer.hashicorp.com/terraform/cli/commands/plan#detailed-exitcode) and [remote/locked S3 backends](https://developer.hashicorp.com/terraform/language/backend/s3). | ||
| - [GitHub OIDC](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect) for keyless cloud authentication from Actions. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Improvement: The redirect from the intermediate URL (
/tutorials/terraform_drift_detection) lands on the "non-Terraform changes" page. A reader who bookmarked the old single-page tutorial might expect to see the statefile-provenance content too. Consider whether a short landing page or a redirect to a parent group would be friendlier — or at minimum, the target page's intro already cross-links to the sibling, which mitigates this. Noting for awareness.