Skip to content

feat: add EOL package archival and pruning tooling#191

Open
abtreece wants to merge 5 commits into
fullstaq-ruby:mainfrom
abtreece:feat/eol-archive-repos
Open

feat: add EOL package archival and pruning tooling#191
abtreece wants to merge 5 commits into
fullstaq-ruby:mainfrom
abtreece:feat/eol-archive-repos

Conversation

@abtreece
Copy link
Copy Markdown
Collaborator

@abtreece abtreece commented Apr 25, 2026

Summary

  • Add incremental APT/YUM archive migration scripts that move EOL distro packages to archive buckets
  • Add APT/YUM pruning scripts that remove EOL Ruby version packages from still-supported distro repos
  • Add comprehensive dev-handbook runbook for the archival process
  • Update dev-handbook docs with accurate bucket names and archive architecture

Context

Addresses #190 — CI disk space exhaustion caused by the ever-growing Aptly state.tar.zst.

Two-axis cleanup:

  1. Distro archival — Move entire EOL distro repos (centos-7, centos-8, debian-9, debian-10, ubuntu-18.04, ubuntu-20.04) to separate archive GCS buckets
  2. Package pruning — Remove EOL Ruby version packages (2.5-3.1) from still-supported distro repos

Migration scripts are incremental — safe to re-run when future distros reach EOL. Each run fetches the existing archive state, merges new content, and uploads as version N+1.

Retiring centos-8 from the active build matrix (and any other EOL distro) is intentionally out of scope for this PR and will be handled as a follow-up that edits config.yml and regenerates workflows. This PR only delivers the tooling.

Companion PR: fullstaq-ruby/infra#47 (GCS buckets, DNS, Caddy config)
Implementation plan: context/plans/EOL-ARCHIVE-MIGRATION.md (Tasks 4-9)

New scripts

Script Purpose
archive/migrate-apt-to-archive.rb Move EOL distro APT repos to archive bucket
archive/migrate-yum-to-archive.rb Move EOL distro YUM repos to archive bucket
archive/prune-apt-packages.rb Remove EOL Ruby packages from APT state
archive/prune-yum-packages.rb Remove EOL Ruby packages from YUM state

All scripts support --dry-run and auto-detect EOL content by comparing repo state against config.yml.

Test plan

  • eclint check passes
  • Migration scripts: --dry-run correctly identifies EOL distros/packages
  • Migration scripts: incremental archive works (second run merges, doesn't overwrite)
  • Pruning scripts: only fullstaq-ruby-X.Y* packages for non-active minor versions are removed
  • Runbook procedures in dev-handbook/archiving-eol-packages.md match actual script behavior

Add incremental APT/YUM archive migration scripts that move EOL distro
packages to archive buckets, and pruning scripts that remove EOL Ruby
version packages from still-supported distro repos.

Scripts auto-detect EOL content by comparing repo state against
config.yml and support --dry-run. Migration scripts are incremental —
safe to re-run when future distros reach EOL.

Includes dev-handbook runbook and updated architecture docs.
@abtreece abtreece force-pushed the feat/eol-archive-repos branch from 7786b8f to e523f8a Compare April 25, 2026 03:12
- Wire up GCloudStorageLock in all four archive scripts so migrations
  and prunes mutually exclude with the publish workflow's pointer
  updates. The lock library was already imported but never used.
- Wrap all four scripts in begin/ensure cleanup so a failure mid-run
  no longer leaves the imported GPG signing key on disk.
- migrate-apt-to-archive: stop swallowing pool-merge errors via
  '|| true'. Real I/O/permission failures must surface; only
  no-clobber skips were ever benign.
- migrate-apt-to-archive / migrate-yum-to-archive: comments claimed
  EOL detection compared against environments/, but the code compares
  against config.yml. Align the comments with the code.
- archiving-eol-packages.md: APT explicit-distros example listed
  centos-8, which is RPM-only and would fail in the APT script.
  Replace with debian-10,ubuntu-20.04.
@abtreece abtreece force-pushed the feat/eol-archive-repos branch from 6c19add to 68b8e37 Compare April 28, 2026 03:02
@abtreece abtreece requested review from FooBarWidget and rus-max May 15, 2026 05:32
@noahssarcastic
Copy link
Copy Markdown
Collaborator

Are we planning on announcing the removal of EOL Ruby packages in some way so that users can archive them themselves if they are still unmigrated?

Copy link
Copy Markdown
Member

@FooBarWidget FooBarWidget left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of pruning EOL packages, we should move them to the archive repos. EOL packages should still be installable.

Copy link
Copy Markdown
Collaborator Author

@abtreece abtreece left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — installability of EOL Ruby packages on still-supported distros is a guarantee we shouldn't break. The current "pruning" approach is wrong for the same reason we archive EOL distros instead of deleting them.

I'll rework this:

  • prune-apt-packages.rb / prune-yum-packages.rb → archive-move semantics (copy to archive bucket, then remove from live repo, idempotent, dry-run preserved). Rename to archive-* so the tooling name matches what it does.
  • Update dev-handbook/archiving-eol-packages.md and the PR title/description to reflect archival, not pruning.
  • Coordinate with fullstaq-ruby/infra#47 on bucket layout.

One clarifier before I start: should EOL-Ruby-on-supported-distro packages land in the existing per-distro archive buckets (extending their scope so each archive bucket can host EOL versions for both EOL and still-supported distros), or do you want a separate archive layout that mirrors live distros? The first is simpler; the second keeps "EOL distro archive" and "EOL version archive" as distinct concepts. Either way the answer shapes infra#47, so I'd like to settle it before reworking the scripts.

Also re: @noahssarcastic's question above — with archive-instead-of-delete, no user action is required to retain installability, but I'll still add a heads-up to release notes when this ships.

infra#47 renames the archive buckets from `-repo-archive` to
`-archive-repo` for naming consistency with the live buckets
(see fullstaq-ruby/infra@c1814eb).

Scripts here are bucket-name agnostic (they read
`ENV['ARCHIVE_REPO_BUCKET_NAME']`), but the old names were baked into
the runbook examples, the architecture doc, and the script header
docstrings -- updated here so the runbook stays correct after both
PRs merge.

No code changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants