feat: add EOL package archival and pruning tooling#191
Conversation
dbb17ef to
7786b8f
Compare
Add incremental APT/YUM archive migration scripts that move EOL distro packages to archive buckets, and pruning scripts that remove EOL Ruby version packages from still-supported distro repos. Scripts auto-detect EOL content by comparing repo state against config.yml and support --dry-run. Migration scripts are incremental — safe to re-run when future distros reach EOL. Includes dev-handbook runbook and updated architecture docs.
7786b8f to
e523f8a
Compare
- Wire up GCloudStorageLock in all four archive scripts so migrations and prunes mutually exclude with the publish workflow's pointer updates. The lock library was already imported but never used. - Wrap all four scripts in begin/ensure cleanup so a failure mid-run no longer leaves the imported GPG signing key on disk. - migrate-apt-to-archive: stop swallowing pool-merge errors via '|| true'. Real I/O/permission failures must surface; only no-clobber skips were ever benign. - migrate-apt-to-archive / migrate-yum-to-archive: comments claimed EOL detection compared against environments/, but the code compares against config.yml. Align the comments with the code. - archiving-eol-packages.md: APT explicit-distros example listed centos-8, which is RPM-only and would fail in the APT script. Replace with debian-10,ubuntu-20.04.
6c19add to
68b8e37
Compare
|
Are we planning on announcing the removal of EOL Ruby packages in some way so that users can archive them themselves if they are still unmigrated? |
FooBarWidget
left a comment
There was a problem hiding this comment.
Instead of pruning EOL packages, we should move them to the archive repos. EOL packages should still be installable.
abtreece
left a comment
There was a problem hiding this comment.
Agreed — installability of EOL Ruby packages on still-supported distros is a guarantee we shouldn't break. The current "pruning" approach is wrong for the same reason we archive EOL distros instead of deleting them.
I'll rework this:
prune-apt-packages.rb/prune-yum-packages.rb→ archive-move semantics (copy to archive bucket, then remove from live repo, idempotent, dry-run preserved). Rename toarchive-*so the tooling name matches what it does.- Update
dev-handbook/archiving-eol-packages.mdand the PR title/description to reflect archival, not pruning. - Coordinate with
fullstaq-ruby/infra#47on bucket layout.
One clarifier before I start: should EOL-Ruby-on-supported-distro packages land in the existing per-distro archive buckets (extending their scope so each archive bucket can host EOL versions for both EOL and still-supported distros), or do you want a separate archive layout that mirrors live distros? The first is simpler; the second keeps "EOL distro archive" and "EOL version archive" as distinct concepts. Either way the answer shapes infra#47, so I'd like to settle it before reworking the scripts.
Also re: @noahssarcastic's question above — with archive-instead-of-delete, no user action is required to retain installability, but I'll still add a heads-up to release notes when this ships.
infra#47 renames the archive buckets from `-repo-archive` to `-archive-repo` for naming consistency with the live buckets (see fullstaq-ruby/infra@c1814eb). Scripts here are bucket-name agnostic (they read `ENV['ARCHIVE_REPO_BUCKET_NAME']`), but the old names were baked into the runbook examples, the architecture doc, and the script header docstrings -- updated here so the runbook stays correct after both PRs merge. No code changes.
Summary
Context
Addresses #190 — CI disk space exhaustion caused by the ever-growing Aptly
state.tar.zst.Two-axis cleanup:
Migration scripts are incremental — safe to re-run when future distros reach EOL. Each run fetches the existing archive state, merges new content, and uploads as version N+1.
Retiring centos-8 from the active build matrix (and any other EOL distro) is intentionally out of scope for this PR and will be handled as a follow-up that edits
config.ymland regenerates workflows. This PR only delivers the tooling.Companion PR: fullstaq-ruby/infra#47 (GCS buckets, DNS, Caddy config)
Implementation plan:
context/plans/EOL-ARCHIVE-MIGRATION.md(Tasks 4-9)New scripts
archive/migrate-apt-to-archive.rbarchive/migrate-yum-to-archive.rbarchive/prune-apt-packages.rbarchive/prune-yum-packages.rbAll scripts support
--dry-runand auto-detect EOL content by comparing repo state againstconfig.yml.Test plan
eclint checkpasses--dry-runcorrectly identifies EOL distros/packagesfullstaq-ruby-X.Y*packages for non-active minor versions are removeddev-handbook/archiving-eol-packages.mdmatch actual script behavior