fix: pin airflow redis image to the bitnamilegacy archive

[arnoldcastro5000]

Fixes #321. Related to #315 (the tile server investigation) and #320 (metrics-server, which failed for the same root cause).

Background: why Airflow has been down since June 13

Airflow uses Redis as its message queue: the scheduler puts tasks into Redis and the workers take them out. No Redis means no tasks run at all.

On June 13 the cluster nodes were rebuilt and the Redis pod was rescheduled onto a fresh node, which had to pull the Redis container image again. The pull fails with ErrImagePull, because Bitnami (the image publisher) delisted its free public images in 2025. The image the cluster asks for, public.ecr.aws/bitnami/redis:5.0.14-debian-10-r32, no longer exists at that address. Since then the scheduler has restarted 57+ times (its health check fails without a working queue) and the workers 550+ times. Every DAG has been stopped for three weeks.

This is the same event that broke metrics-server (fixed in #320): two casualties of the one Bitnami delisting.

The change

One block added to the Airflow Helm values in the Ansible playbook, pinning the Redis image explicitly:

redis:
  image:
    registry: docker.io
    repository: bitnamilegacy/redis
    tag: "5.0.14"

• bitnamilegacy is Bitnami's official read-only archive on Docker Hub, created for exactly this situation: it holds the delisted images.
• Tag availability was verified against the Docker Hub API on 2026-07-03. The archive does not carry the old suffixed tag (5.0.14-debian-10-r32), but it carries the bare 5.0.14 version alias, which Bitnami used as the rolling name for the newest rebuild of that version. Same Redis 5.0.14 that production ran, so no version change and no upgrade risk (and the queue data is disposable regardless: the chart runs this Redis without persistence).
• The playbook previously set no image at all, so the deployment inherited a default that no longer exists anywhere pullable. Pinning it explicitly also protects us if defaults move again.

The value structure (redis.image.registry/repository/tag) was verified against the chart source: airflow chart 8.5.2 embeds the stable/redis 10.5.7 chart, whose image settings are exactly these three fields.

Why not migrate to an upstream image, like #320 did for metrics-server?

Deliberately different strategies for the two Bitnami casualties:

• metrics-server is a standalone component with a maintained first-party home (the Kubernetes SIG chart and registry.k8s.io images), so Fix metrics-server: migrate from retired Bitnami chart to upstream #320 could move to the canonical source and be done with Bitnami entirely.
• This Redis is not standalone. It is a subchart baked into the Airflow chart, and that subchart's startup scripts are welded to the Bitnami image layout: the pod boots by copying config from /opt/bitnami/redis/mounted-etc/, starts redis with --include /opt/bitnami/redis/etc/redis.conf, and reads its password from /opt/bitnami/redis/secrets/redis-password (verified in the chart templates). The official redis image on Docker Hub has none of those paths, so swapping it in would crashloop. Only a Bitnami-layout image works here, and bitnamilegacy is the archive of exactly those.

The durable escape routes exist but are much bigger changes than a 3 week outage should wait for: either externalRedis.* (run our own Redis outside the chart, on any image we choose) or an Airflow chart upgrade. Both are worth considering later; this PR is the minimal fix that brings Airflow back. Being frozen is acceptable for this image because Redis 5.0.14 was already frozen (it is past end of life); the real modernization path is the chart upgrade, tracked separately.

IMPORTANT: do not deploy this before node-mapnik-1#113

Restoring Airflow restarts the map pre-warm DAGs, which send the heaviest database queries in the system to the tile server every 5 minutes. The tile server currently has no query time limit; that combination is what caused the original #315 incident. Two protections must be in place first:

1. fix: kill hung queries at 25 seconds and stop tile requests hanging forever on database errors treetracker-tile-server#113 (adds a 30 second query timeout to the tile server) must be merged AND deployed.
2. The gentler retry settings for the pre-warm DAGs (companion PR in treetracker-airflow-dags) should be merged to the production branch, so they are in force the moment Airflow wakes up.

Merging THIS PR is safe at any time: the playbook change does nothing until someone runs the Ansible playbook against production.

Checks to run at deploy time

1. Final pull check right before deploy: docker manifest inspect docker.io/bitnamilegacy/redis:5.0.14 (the tag's existence was already confirmed via the Hub API on 2026-07-03; this re-confirms nothing moved). If the archive is ever removed, the fallback is to mirror the image into the greenstand Docker Hub organization and point this block there (a one line change).
2. Compare the live release values with this playbook before running Ansible: helm get values airflow -n airflow. The dead pod referenced an image this playbook never set, so the live release contains overrides the playbook does not know about; make sure running the playbook does not silently revert something else.
3. After deploy: the redis pod reaches Running, the scheduler restart count stops climbing, workers stabilize, and the pre-warm DAGs show the new retry settings in the UI before anything is unpaused.

Follow-up (not in this PR)

The Redis pod runs with no resource requests or limits, which puts it first in line for eviction under node memory pressure (the same weakness the tile server had, see #315). We deliberately did not guess values here: after restoration the pod will be measured under real load for a day or two, and a follow-up PR will add measured requests and limits.

[arnoldcastro5000]

Deployed

bitnamilegacy/redis:5.0.14

to dev. Will monitor before deploying to prod.

[arnoldcastro5000]

Airflow redis restored in prod and running for 35h.