Skip to content

feat: add archive infrastructure for EOL distribution packages #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
abtreece wants to merge 5 commits into
base: main
Choose a base branch
from
+149 −3
Open

feat: add archive infrastructure for EOL distribution packages #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d23875c
feat: add GCS buckets for APT/YUM archive repositories
abtreece Apr 25, 2026
cb9de39
feat: add DNS zones for APT/YUM archive subdomains
abtreece Apr 25, 2026
7e7c90e
feat: add Caddy config and version query for archive repos
abtreece Apr 25, 2026
4f829ce
fix: archive version lookup must fail hard on non-404 errors
abtreece Apr 28, 2026
b256946
Merge branch 'main' into feat/eol-archive-repos
FooBarWidget May 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions ansible/files/Caddyfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,35 @@ yum.{$DOMAIN_NAME} {
redir https://storage.googleapis.com/{$GCLOUD_BUCKET_PREFIX}-server-edition-yum-repo/versions/{$YUM_LATEST_VERSION}/public{uri}
}
}

apt-archive.{$DOMAIN_NAME} {
tls {
dns azure {
tenant_id {$AZURE_TENANT_ID}
subscription_id {$AZURE_SUBSCRIPTION_ID}
resource_group_name fullstaq-ruby-infra-maintainers
client_id {$AZURE_DNS_UPDATER_CLIENT_ID}
client_secret {$AZURE_DNS_UPDATER_CLIENT_SECRET}
}
}
encode gzip
handle {
redir https://storage.googleapis.com/{$GCLOUD_BUCKET_PREFIX}-server-edition-apt-repo-archive/versions/{$APT_ARCHIVE_LATEST_VERSION}/public{uri}
}
}

yum-archive.{$DOMAIN_NAME} {
tls {
dns azure {
tenant_id {$AZURE_TENANT_ID}
subscription_id {$AZURE_SUBSCRIPTION_ID}
resource_group_name fullstaq-ruby-infra-maintainers
client_id {$AZURE_DNS_UPDATER_CLIENT_ID}
client_secret {$AZURE_DNS_UPDATER_CLIENT_SECRET}
}
}
encode gzip
handle {
redir https://storage.googleapis.com/{$GCLOUD_BUCKET_PREFIX}-server-edition-yum-repo-archive/versions/{$YUM_ARCHIVE_LATEST_VERSION}/public{uri}
}
}
19 changes: 16 additions & 3 deletions ansible/files/query-latest-repo-versions.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,15 @@ def main
open_io(ARGV[0]) do |io|
query_repo_version('apt', io)
query_repo_version('yum', io)
query_repo_version('apt-archive', io, suffix: '-archive')
query_repo_version('yum-archive', io, suffix: '-archive')
io.puts "REPO_QUERY_TIME=#{Time.now.to_f}"
end
end

def query_repo_version(type, output)
uri = URI("https://storage.googleapis.com/#{require_env(:GCLOUD_BUCKET_PREFIX)}-server-edition-#{type}-repo/versions/latest_version.txt")
def query_repo_version(type, output, suffix: '')
Copy link
Copy Markdown
Collaborator

@noahssarcastic noahssarcastic May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be a small nit but I find the subtractive nature of the suffix parameter a little confusing. I think it would be slightly clearer if a call with a suffix was written as: query_repo_version('yum', io, suffix: '-archive'). Not going to mark this as blocking though :)

Copy link
Copy Markdown
Member

@FooBarWidget FooBarWidget May 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seconded.

bucket_type = type.gsub('-archive', '')
uri = URI("https://storage.googleapis.com/#{require_env(:GCLOUD_BUCKET_PREFIX)}-server-edition-#{bucket_type}-repo#{suffix}/versions/latest_version.txt")

STDERR.puts "Querying #{uri}..."
req = Net::HTTP::Get.new(uri)
Expand All @@ -22,11 +25,21 @@ def query_repo_version(type, output)
end

if resp.code.to_i / 100 != 2
# Archive bucket may legitimately not exist before the first migration runs.
# Only treat 404 as the "not yet populated" case; any other non-2xx
# (auth, 5xx, redirects) is a real failure and must surface — silently
# falling back to version 0 would point clients at /versions/0/... 404s.
if suffix != '' && resp.code.to_i == 404
STDERR.puts "Warning: #{type} repo not found (404), skipping"
output.puts "#{type.upcase.gsub('-', '_')}_LATEST_VERSION=0"
return
end
abort("Failed to query #{uri}: #{resp.code} #{resp.body}")
end

env_key = type.upcase.gsub('-', '_')
STDERR.puts "#{type} latest version: #{resp.body}"
output.puts "#{type.upcase}_LATEST_VERSION=#{resp.body}"
output.puts "#{env_key}_LATEST_VERSION=#{resp.body}"
end

def open_io(path)
Expand Down
72 changes: 72 additions & 0 deletions terraform/dns.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,3 +149,75 @@ resource "azurerm_dns_aaaa_record" "yum" {
records = [var.backend_server_ipv6]
ttl = 86400
}


resource "azurerm_dns_zone" "apt-archive" {
name = "apt-archive.${var.dns_name}"
resource_group_name = "fullstaq-ruby-infra-maintainers"
}

resource "azurerm_role_assignment" "caddy-update-dns-apt-archive" {
scope = azurerm_dns_zone.apt-archive.id
role_definition_name = "DNS Zone Contributor"
principal_id = azuread_service_principal.caddy.object_id
}

resource "azurerm_dns_ns_record" "apt-archive" {
name = "apt-archive"
zone_name = azurerm_dns_zone.website.name
resource_group_name = azurerm_dns_zone.website.resource_group_name
ttl = 86400
records = azurerm_dns_zone.apt-archive.name_servers
}

resource "azurerm_dns_a_record" "apt-archive" {
name = "@"
zone_name = azurerm_dns_zone.apt-archive.name
resource_group_name = azurerm_dns_zone.apt-archive.resource_group_name
records = [var.backend_server_ipv4]
ttl = 86400
}

resource "azurerm_dns_aaaa_record" "apt-archive" {
name = "@"
zone_name = azurerm_dns_zone.apt-archive.name
resource_group_name = azurerm_dns_zone.apt-archive.resource_group_name
records = [var.backend_server_ipv6]
ttl = 86400
}


resource "azurerm_dns_zone" "yum-archive" {
name = "yum-archive.${var.dns_name}"
resource_group_name = "fullstaq-ruby-infra-maintainers"
}

resource "azurerm_role_assignment" "caddy-update-dns-yum-archive" {
scope = azurerm_dns_zone.yum-archive.id
role_definition_name = "DNS Zone Contributor"
principal_id = azuread_service_principal.caddy.object_id
}

resource "azurerm_dns_ns_record" "yum-archive" {
name = "yum-archive"
zone_name = azurerm_dns_zone.website.name
resource_group_name = azurerm_dns_zone.website.resource_group_name
ttl = 86400
records = azurerm_dns_zone.yum-archive.name_servers
}

resource "azurerm_dns_a_record" "yum-archive" {
name = "@"
zone_name = azurerm_dns_zone.yum-archive.name
resource_group_name = azurerm_dns_zone.yum-archive.resource_group_name
records = [var.backend_server_ipv4]
ttl = 86400
}

resource "azurerm_dns_aaaa_record" "yum-archive" {
name = "@"
zone_name = azurerm_dns_zone.yum-archive.name
resource_group_name = azurerm_dns_zone.yum-archive.resource_group_name
records = [var.backend_server_ipv6]
ttl = 86400
}
29 changes: 29 additions & 0 deletions terraform/repo_buckets.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,32 @@ resource "google_storage_bucket_iam_binding" "server-edition-yum-repo-writable-b
role = "roles/storage.objectAdmin"
members = ["principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.github-ci-deploy.name}/attribute.repository/fullstaq-ruby/server-edition"]
}

resource "google_storage_bucket" "server-edition-apt-repo-archive" {
depends_on = [google_project_service.storage-api]
name = "${var.gcloud_bucket_prefix}-server-edition-apt-repo-archive"
force_destroy = true
uniform_bucket_level_access = true
location = var.gcloud_storage_location
}

resource "google_storage_bucket_iam_binding" "server-edition-apt-repo-archive-public-viewable" {
bucket = google_storage_bucket.server-edition-apt-repo-archive.self_link
role = "roles/storage.objectViewer"
members = ["allUsers"]
}


resource "google_storage_bucket" "server-edition-yum-repo-archive" {
depends_on = [google_project_service.storage-api]
name = "${var.gcloud_bucket_prefix}-server-edition-yum-repo-archive"
force_destroy = true
uniform_bucket_level_access = true
location = var.gcloud_storage_location
}

resource "google_storage_bucket_iam_binding" "server-edition-yum-repo-archive-public-viewable" {
bucket = google_storage_bucket.server-edition-yum-repo-archive.self_link
role = "roles/storage.objectViewer"
members = ["allUsers"]
}