RFC: Ansible test coverage for the Hetzner playbook (Molecule + Lima)

[abtreece]

RFC: Ansible test coverage for the Hetzner playbook (Molecule + Lima)

Author: @abtreece
Status: Draft for maintainer sign-off
Target repo: fullstaq-ruby/infra (this issue lives in umbrella because it requests cross-cutting policy approval before the PR lands)
Spike branch: abtreece/infra@spike/ansible-molecule-mvp (fork; two commits — 103026a MVP scenario + seam edits, 0a5d120 recurse:true idempotence fix)
Plan + results: posted alongside this RFC on request — full plan is ~660 lines, Phase 0 results is ~150 lines

Summary

I'm proposing we add a Molecule + Lima test scenario to infra/ansible/ so we can converge the Ansible playbook against a real VM locally and in CI. A working MVP has been built and proves the approach end-to-end. Before I open the PR, I'd like maintainer sign-off on the addition. There are no live Azure / GCS / GitHub calls and four small prod-file edits that the test seam requires.

Problem

infra/ansible/ is the sole configuration management for backend.fullstaqruby.org (~351 lines across 11 task files) and has zero automated test coverage. code-reviews.yml validates EditorConfig and Terraform but does not exercise Ansible. Today's loop is: edit → PR → reviewer reads diff → merge → hope it converges on the live VM. That loop misses undefined vars, deprecated module syntax, template render errors, idempotence drift, missing handler notifications, and broken systemd units.

Proposal

Add a single Molecule scenario (ansible/molecule/default/) using the delegated driver against a Lima-managed Debian 12 VM. The scenario:

• Runs main.yml's task list with molecule_test: true, which causes four prod tasks to skip live external calls
• Stages a minimal service-contract fixture (Sinatra+Puma) so the apiserver systemd unit can start
• Asserts via verify.yml that all five services (caddy, prometheus, fail2ban, ssh, apiserver) are systemctl is-active, the apiserver Unix socket exists, caddy validate passes, UFW is active, SSH is hardened, and unattended-upgrade --dry-run succeeds
• Passes molecule idempotence (changed=0 on second converge)

One scenario, one intended source of truth (main.yml) — with Phase 2 drift prevention required because converge.yml currently mirrors the task list (see "Known gaps" below) — one assertion file. Lima as the VM backend both locally and in CI (ubuntu-24.04 runners expose /dev/kvm since Apr 2024).

Proof

Phase 0 (test-seam spike) is complete. molecule test runs the full lifecycle clean end-to-end: destroy → create → prepare → converge → idempotence → verify → destroy. Zero live Azure / GCS / GitHub-release calls. All five services active against the fixture.

Full results doc with exit-criteria checklist, architecture, tooling, and reproduction steps will be attached as a follow-up comment on this issue (collapsed <details> block) so it's one click away rather than a separate request.

What this required (the load-bearing question)

Four prod files needed minimal edits to introduce a molecule_test test seam. Total: ~10 lines added, all behind when: not (molecule_test | default(false)) or a Jinja {% if molecule_test %} guard. Prod behavior is functionally identical when molecule_test is unset — the seam edits alone do not change prod runtime. (Ansible's default trim_blocks: true should make the caddy-env.j2 render byte-identical too, but I'll diff the rendered file against main before opening the productionization PR rather than rely on the default.)

File	Change
templates/caddy-env.j2	Jinja conditional substitutes stub values for the two lookup('pipe', az keyvault ...) calls
tasks/caddy.yml	Gates the repo-version query script and Caddy config copy (Molecule installs stubs at the same paths)
tasks/apiserver-deployer.yml	Gates the deployer script copy (Molecule installs a no-op stub)
tasks/unattended-upgrades.yml	Gates the GitHub-release deb (unattended-upgrades-prometheus-collector)

These four edits are the load-bearing change set this RFC asks maintainers to accept. The MVP collapses to lint-only without them.

CI plan (sketch)

Out of scope for the MVP PR — flagged here so maintainers can sign off on the direction without committing to specifics. Concrete .github/workflows/ansible.yml lands in the productionization PR.

• Runner: ubuntu-24.04 (x64). GitHub-hosted x64 runners expose /dev/kvm since Apr 2024, which makes Lima viable in CI directly. Caveat: GitHub's docs label nested virtualization "experimental and unsupported" even though /dev/kvm is exposed — see fallback below.
• Trigger: path-filtered pull_request and push on ansible/** and the workflow file itself. No always-on cost.
• Two jobs:
  1. lint — yamllint + ansible-lint against ansible/. Cheap, runs first.
  2. molecule — molecule test (full lifecycle) on ubuntu-24.04 against a Lima Debian 12 guest. needs: lint.
• KVM access: the runner user lacks kvm group membership by default; the workflow installs a one-line udev rule (KERNEL=="kvm", GROUP="kvm", MODE="0666") before starting Lima.
• Image cache: actions/cache keyed on lima.yaml hash for ~/.cache/lima (the Lima-actions setup helper does not cache by itself). Cuts steady-state runtime by avoiding a fresh image download per run.
• No secrets: the four seam edits ensure no live Azure / GCS / GitHub-release calls and no secrets are needed, so this is safe on third-party PRs without secret exposure — consistent with the existing code-reviews.yml no-secrets-on-third-party-PRs constraint. (The scenario still needs general internet access for apt, Debian image download, Caddy binary download, and fixture bundle install — it isn't air-gapped.)
• Fallback if Lima-in-CI proves unreliable: held in reserve, not built. Two options: (a) run the same scenario against an ephemeral Hetzner Cloud VM via a CI-only secret (loses the no-secrets property for third-party PRs), or (b) degrade CI to lint-only and keep molecule test as a contributor-only loop. Decision deferred until we have CI runtime data.
• Estimated runtime (needs CI measurement): lint ~30s; molecule ~5–8 min cold (image download + apt installs). Warm-cache numbers are harder to predict — molecule test destroys the VM each run, so apt installs and fixture bundle install rerun even with a cached Lima image. I'll measure once the workflow lands and revise.

Known gaps and Phase 2 commitments

The MVP is deliberately minimal. Before merging the productionization PR (Phase 2 in the plan), I'll close these gaps that surfaced during the spike or in pre-RFC review:

• Drift prevention between main.yml and converge.yml. The converge playbook currently mirrors main.yml's task-import list. That's drift waiting to happen. Phase 2 will either factor the shared task imports into a single included file used by both, or add a small CI check that asserts the two import lists match.
• Caddy reverse-proxy wiring is not actually exercised end-to-end. Today verify.yml checks Caddy is active and caddy validate passes. Phase 2 will add a cheap curl -fsS http://127.0.0.1:8080/admin/health through the test Caddyfile to the apiserver Unix socket — proving the proxy wiring without testing app semantics.
• Production Caddyfile validation. The runtime scenario installs a test Caddyfile (to avoid ACME and pin the proxy route deterministically), which means caddy validate is currently checking the test config — not ansible/files/Caddyfile. A broken prod Caddyfile could pass today. Phase 2 will add a separate caddy validate step against the production Caddyfile rendered with dummy env values, so syntax/adapter regressions in the prod config are still caught.
• Collection pinning. requirements.txt pins Python deps; community.general and ansible.posix are currently installed manually. Phase 2 will add ansible/requirements.yml pinning the collections and wire ansible-galaxy collection install -r requirements.yml into both contributor docs and CI. yamllint and ansible-lint will be pinned alongside (Phase 1).
• Categorize the bootstrap packages. The MVP installs cron, ufw, acl, rsyslog, ruby-dev, build-essential in prepare.yml. These split into three categories: server runtime deps that probably belong in tasks/essentials.yml (cron, ufw, rsyslog); Ansible transport/become deps that legitimately stay in prepare.yml (acl); and fixture-build-only deps that stay in prepare.yml because prod ships pre-built vendor/bundle (ruby-dev, build-essential). I'll classify each before opening the productionization PR.

Bonus: latent prod issues surfaced

Running the MVP against a fresh Debian 12 cloud image surfaced eight real gaps in the playbook, masked in prod by the specific Hetzner image's preinstalled packages or by manual setup history. The most significant:

• tasks/apiserver-deployer.yml is permanently non-idempotent. Create apiserver deployment directory used recurse: true with mode: 0755. The recurse walked into vendor/bundle/ruby/*/bin/ (gem-bin shims) and the latest symlink — both report 0777 from stat() because they're symlinks. Linux can't actually chmod a symlink, but Ansible still reports changed: true. Every ansible-playbook run after the first apiserver release deploy has been reporting at least one task changed, forever. No idempotence test in CI to catch it. Fix is included in the spike branch — recurse: true removed; release tarballs are extracted by the deployer-as-itself, so ownership inside is already correct and the recurse was over-defensive.
• Undeclared package dependencies surfaced on a fresh Debian cloud image. cron, ufw, rsyslog are server-runtime gaps the playbook assumes preinstalled; acl is an Ansible become-as-non-root transport dep; ruby-dev and build-essential are fixture-build-only (prod ships a pre-built vendor/bundle, so they aren't prod gaps). See "Categorize the bootstrap packages" above.
• One ansible_architecture deprecation warning (will break in ansible-core 2.24)

Per the singleton-issue convention, I'd file these as separate focused issues in fullstaq-ruby/infra rather than bundle them into the MVP PR. Open to maintainer preference.

Non-goals (faithful to the plan)

• No real ACME/TLS issuance (test Caddyfile uses auto_https off, listens :8080)
• No real Azure Key Vault, GCS, or GitHub-release integration
• No OIDC JWT verification (covered by infra/apiserver/ Ruby tests)
• No live fail2ban banning (only verifies the unit starts and config parses)
• No AppArmor profile management (today's task only installs the package)

Questions for maintainers

1. Are the four molecule_test-gated edits in prod task files acceptable? They are no-ops in prod but introduce a test-mode concept into the playbook. This is the load-bearing approval the rest of the work depends on.
2. Should bootstrap packages (cron, ufw, acl, rsyslog) be added to the production playbook (e.g. tasks/essentials.yml) so it's self-contained on a bare Debian image? The MVP installs them in Molecule's prepare.yml, preserving prod behavior but documenting the dependency.
3. Bundle vs split the latent prod issues? Recommend: split (one issue per finding). Confirm preference.
4. Two PRs, not one. I'll split the spike branch into (a) MVP scenario + the four seam edits and (b) the recurse: true idempotence fix. Distinct intent, distinct risk profile — either could land independently. Flagging in case a maintainer prefers a different grouping.
5. Is the CI shape sketched above acceptable? Specifically: GitHub-hosted ubuntu-24.04 runners with Lima (vs. a self-hosted runner or a Hetzner-cloud fallback), and no-secrets-on-third-party-PRs preserved by the offline-friendly seam design. Direction-only sign-off is enough; concrete workflow lands in the productionization PR.

Proposed next steps

If maintainers approve the four test-seam edits:

1. Open PR (a) against fullstaq-ruby/infra with the MVP scenario + the four seam edits
2. Open PR (b) with the recurse: true idempotence fix (independent; can land in either order)
3. File separate issues for the remaining latent findings (one per finding)
4. Phases 1–4 from the plan (lint, productionize with the gaps above closed, CI integration, announce) — follow-up PRs

[abtreece]

Phase 0 — Test-Seam Spike Results (click to expand)

Phase 0 — Test-Seam Spike Results

Status: Complete (hypothesis proven)
Date: 2026-04-30
Branch: spike/ansible-molecule-mvp in infra/
Worktree: infra/.worktrees/ansible-molecule-mvp/
Plan reference: PLAN.md §"Phase 0 — Test-seam spike"
Effort: ~1 day, single contributor

Outcome

molecule converge runs end-to-end against a Lima VM with zero live Azure
Key Vault, GCS, or GitHub-release network calls, and all five target services
(caddy, prometheus, fail2ban, ssh, apiserver) reach systemctl is-active = active against a service-contract fixture. Verify and idempotence
both pass.

This proves the load-bearing claim from the plan: the same playbook can run in
prod and in a Molecule scenario, with the only divergence being four narrowly
scoped molecule_test guards in prod files plus a separate Molecule converge
playbook that mirrors main.yml's task list.

Phase 0 exit criteria — all met

Criterion	Result	Evidence
caddy active	✅	verify.yml — Core services are active task
prometheus active	✅	same
fail2ban active	✅	same
ssh active	✅	same
apiserver active	✅	same
apiserver-deployer one-shot active (RemainAfterExit)	✅	verify.yml
Apiserver Unix socket exists at /run/apiserver/server.sock	✅	verify.yml stat task
caddy validate passes against installed config	✅	verify.yml
UFW active with rules	✅	verify.yml
SSH hardened (PermitRootLogin without-password)	✅	verify.yml assert
unattended-upgrade --dry-run succeeds	✅	verify.yml
Zero Azure / GCS / GitHub-release calls	✅ (by inspection)	All four seams gated; verified by code inspection of the gated tasks and by the absence of those external endpoints in the converge log. Note: this is evidence-by-inspection, not a stronger deny-by-default proof (e.g. egress-blocked guest or strace-on-controller). A future hardening step could enforce this at the network layer rather than relying on inspection.
Idempotence (changed=0 on second converge)	✅	molecule idempotence (full molecule test lifecycle clean end-to-end)

Test-seam refactors landed (least-invasive)

Four prod files touched. Total: ~10 lines added, all behind
when: not (molecule_test | default(false)) or a Jinja {% if molecule_test %}
guard, so prod behavior is byte-identical when molecule_test is unset —
the seam edits alone do not change prod runtime.

A separate prod-behavior change also landed in this branch (the recurse: true
removal in tasks/apiserver-deployer.yml — see finding #6 below). That fix
does alter prod runtime: it eliminates a permanent idempotence flap. It is
distinct from the seam edits in both intent and risk profile and may belong in
a separate PR (see "Recommended next steps").

File	Change
templates/caddy-env.j2	Jinja conditional substitutes stub values for the two lookup('pipe', az keyvault ...) calls when molecule_test is true
tasks/caddy.yml	Gates "Install repo version query script" and "Install Caddy config file" — Molecule installs stubs at the same paths during converge
tasks/apiserver-deployer.yml	Gates "Install apiserver-deployer" script copy — Molecule installs a no-op stub
tasks/unattended-upgrades.yml	Gates the GitHub-release deb (unattended-upgrades-prometheus-collector) — pure dependency removal in test mode

These four diffs are the load-bearing change set the RFC asks maintainers to
accept. Diff in the spike branch: git diff main -- ansible/templates/caddy-env.j2 ansible/tasks/caddy.yml ansible/tasks/apiserver-deployer.yml ansible/tasks/unattended-upgrades.yml.

Architecture as built

infra/ansible/
├── main.yml                          # unchanged
├── tasks/                            # 3 files modified (when: guards only)
├── templates/caddy-env.j2            # 1 Jinja conditional added
├── files/
│   └── molecule-test/                # NEW — test-only artifacts colocated with prod files
│       ├── stub-query-latest-repo-versions
│       ├── stub-apiserver-deployer
│       ├── test-Caddyfile            # auto_https off, listens :8080, proxies admin to socket
│       └── apiserver-fixture/        # minimal Sinatra app for the service-contract fixture
│           ├── Gemfile
│           └── config.ru
├── requirements.txt                  # NEW — pinned molecule, ansible-core, molecule-plugins
└── molecule/
    └── default/
        ├── molecule.yml              # delegated driver, ansible verifier
        ├── lima.yaml                 # Debian 12 cloud image, native arch
        ├── create.yml                # limactl start + write inventory with Lima SSH config
        ├── destroy.yml               # limactl delete --force
        ├── prepare.yml               # bootstrap pkgs the playbook assumes preinstalled (see findings)
        ├── converge.yml              # mirrors main.yml task list + injects test-only fixture/stub tasks
        ├── verify.yml                # systemd / socket / caddy validate / ufw / sshd -T / dry-run
        ├── files -> ../../files      # symlink so import_tasks copy: src lookups resolve to ansible/files/
        └── templates -> ../../templates

The files/templates symlinks were necessary because Ansible's relative-path
search for copy: src: and template: src: resolves against the playbook
directory. Since converge.yml lives in molecule/default/, those paths would
not find ansible/files/ without the symlinks.

Latent prod issues surfaced during the spike

These were real failures during the first converge attempts on a fresh Debian
12 cloud image. Each is a real gap in the playbook, masked in prod by the
specific Hetzner image's preinstalled packages or by manual setup history.
Including them is the most concrete demonstration of the MVP's value — and
finding #6 in particular was a permanent prod-side idempotence bug that this
branch also fixes:

1. cron package not declared. tasks/autoreboot.yml uses ansible.builtin.cron: which requires crontab on PATH. Fresh Debian cloud image has no cron.
2. ufw package not declared. Whole tasks/ufw.yml fails on a fresh VM.
3. acl package not declared. Any become_user: to a non-root user fails (Ansible needs setfacl to share temp files with unprivileged users).
4. rsyslog not declared. fail2ban sshd jail fails to start: "Have not found any log file for sshd jail" (Debian cloud image uses systemd journal only).
5. ruby-dev / build-essential not declared. Any gem with native extensions fails to compile. Masked in prod because the apiserver release tarball ships pre-built vendor/bundle.
6. tasks/apiserver-deployer.yml:28 was permanently non-idempotent (FIXED in this branch). The Create apiserver deployment directory task used recurse: true with mode: 0755. The recurse walked into vendor/bundle/ruby/*/bin/ (gem-bin shims) and the latest symlink — both report 0777 from stat() because they're symlinks. Linux can't actually chmod a symlink, but Ansible still reports changed: true because requested mode ≠ observed mode. In prod this means every ansible-playbook run after the first apiserver release deploy has been reporting at least one task changed, forever. Nobody noticed because there was no idempotence test in CI. Fixed by removing recurse: true — release tarballs are extracted by the deployer running as itself, so file ownership inside is already correct, and the recurse was over-defensive.
7. tasks/caddy.yml:9 deprecation warning. References ansible_architecture directly; will break in ansible-core 2.24 per the warning.
8. tasks/caddy.yml:1 get_url has no explicit timeout. caddyserver.com builds custom-asset binaries on demand and exceeded the default 10s timeout intermittently.

For Phase 0 the bootstrap (cron, ufw, acl, rsyslog, ruby-dev, build-essential)
is installed by molecule/default/prepare.yml, deferring the question of
whether to also declare them in the production playbook — that's a maintainer
discussion, but the data is now available.

What's not in scope here

Faithful to the plan's non-goals:

• No real ACME/TLS issuance (test Caddyfile uses auto_https off and listens :8080)
• No real Azure Key Vault, GCS, or GitHub-release calls
• No OIDC JWT verification (covered by infra/apiserver/ Ruby tests)
• No live fail2ban banning (only verifies the unit starts and config parses)
• AppArmor profile management (today's task only installs the package)

Tooling

• lima 2.1.1, limactl
• molecule 26.4.0 + molecule-plugins 25.8.12 + ansible-core 2.20.5 (installed via uv tool install)
• community.general and ansible.posix collections (installed via ansible-galaxy)
• Lima VM: Debian 12 cloud image, native arch (arm64 on Apple Silicon contributor laptops)

Pinned in infra/ansible/requirements.txt.

How to reproduce

See infra/ansible/README.md (added in this branch).

Recommended next steps for the RFC

1. Present this branch + this document to maintainers.
2. Ask for sign-off on the four test-seam refactors (the load-bearing prod-file changes). Without those, the MVP collapses to lint-only (Phase 1 of the plan).
3. Split the spike branch into two PRs: (a) MVP scenario + the four seam edits; (b) the recurse: true idempotence fix. Distinct intent, distinct risk. Either could land independently.
4. Decide whether the remaining latent prod issues (1)–(5), (7), (8) should be filed as individual focused issues (recommend: split — one issue per finding, per the singleton-issue convention).
5. If maintainers approve seams, proceed with PLAN.md Phases 1 (lint), 2 (productionize), 3 (CI integration), 4 (announce). Phase 2 should now also include: drift-prevention between main.yml and converge.yml, an end-to-end Caddy → Unix-socket health check, and an ansible/requirements.yml for collection pinning.

Open questions for maintainer review

• Are the four molecule_test-gated edits acceptable in prod files? They are no-ops in prod, but they introduce a test-mode concept into the playbook. The plan's "Open Question #2" explicitly flagged this as needing pre-approval.
• Should the bootstrap packages (cron, ufw, acl, rsyslog) be added to the production playbook (e.g. tasks/essentials.yml) so the playbook is self-contained on a bare Debian image? The MVP has them only in Molecule's prepare.yml, which preserves prod behavior but documents the dependency.
• Is ansible/files/molecule-test/ an acceptable home for test-only artifacts shipping inside the prod files dir? Alternative is keeping them in ansible/molecule/default/files/ proper, but that requires the symlink workaround (already in use here for prod files).