Skip to content

feat!: regional-only deployment — drop legacy append-user support - #33

Merged
venkatamutyala merged 10 commits into
mainfrom
feat/regional-tunnel-endpoints
Aug 2, 2026
Merged

feat!: regional-only deployment — drop legacy append-user support#33
venkatamutyala merged 10 commits into
mainfrom
feat/regional-tunnel-endpoints

Conversation

@venkatamutyala

@venkatamutyala venkatamutyala commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

What

Makes this repo regional-only: the stack deploys per-datacenter tunnel servers for the regional CDE rollout, and legacy support is removed rather than parameterized (an earlier revision of this PR had a SISH_SUBDOMAIN_FLAGS toggle with a legacy default — dropped by decision).

  • --force-requested-subdomains is hardcoded: a codespace VM binds its bare hostname, URLs are https://<hostname>.<region>.tunnels.cde.glueopshosted.com (no prefix), and a taken name fails the bind instead of silently going random. The append-user flags are gone.
  • .env.example targets a regional deployment: DOMAIN=<region>.tunnels.cde.glueopshosted.com, AWS creds from the acme-dns01-cde.glueopshosted.com IAM user (cde_acme_* outputs of the CDE assets workspace).
  • README documents the regional model, the CloudFront relationship (this box is the origin as origin.<region>..., covered by the same wildcard cert; 443 deliberately stays directly reachable), and a smoke test that catches a wrong-version deployment by URL shape.
  • MIGRATION.md (old-fork → this stack, central-box content) is removed; it remains in git history and pre-0.4 tags.

The legacy central box (tunnels.glueopshosted.com) is unaffected — it stays on its current pre-0.4 checkout and needs no changes.

Also folded in: Renovate updates + auth hardening

Supersedes the open Renovate PRs (#28, #29, #30): base image → python:3.14-alpine (taking #29 over #28's 3.13) and click==8.4.2. A Python-expert review verified every pin on the exact digest (cp314 musllinux wheels present with MarkupSafe's C speedups active; Flask/Werkzeug CI-test 3.14 upstream; gunicorn 26.0.0 lacks official 3.14 metadata but its sync-worker path is exercised and passing) — Renovate will auto-close those PRs once this merges.

The same review found three live-reproducible defects in the TOFU authorizer, fixed here in auth/auth.py:

  • a non-string auth_key crashed mid-registration and left an empty key file, permanently locking that username out — JSON shape/types are now validated up front;
  • distinct raw usernames could collapse onto one key file via secure_filename ("vm 1" vs "vm_1") — any username the sanitizer would alter is now rejected outright (VM hostnames always pass unchanged);
  • registration had a create-then-write window exposing empty files to concurrent workers — keys are now written to a temp file and hard-linked into place atomically, and a same-key race loser (same VM connecting twice at boot) is still allowed.

Plus: 16 KB request cap, remote_addr in the audit log lines, and PYTHONUNBUFFERED=1 so those lines actually flush. Full flow re-tested in the built image: register/allow/deny, poisoning attempt then legitimate registration, collision rejection, path traversal, no temp/empty leftovers, oversized body → 413.

A second review pass (fresh Python expert + a sish expert working from the tagged v2.23.0 source) confirmed the wire contract exactly (payload shape, comment-free/newline-free key format, per-key deny semantics, TOFU re-checked on every reconnect, adversarial 32-way race tests producing exactly one registration) and added:

  • auth.py: usernames capped at the 63-char hostname label limit (oversized attacker-chosen SSH usernames previously reached the filesystem → traceback-spamming 500s from the public SSH port); any OSError reading a stored key now denies instead of 500ing; whitespace-only keys rejected.
  • compose: --authentication-key-request-timeout 15s → 3s — sish holds a global lock across the webhook call, so a hung authenticator serialized all auth on the instance; 3s bounds the stall (the webhook answers in ms). --authentication-keys-directory now points at a tmpfs, making sish's webhook-bypass path (any key file in that directory is accepted without consulting the authenticator) empty by construction.
  • companion fix in fix: pin sish tunnel auth to the dedicated key with IdentitiesOnly codespaces#551: -o IdentitiesOnly=yes on the autossh invocation — without it, a forwarded ssh-agent's keys are offered first and TOFU can permanently pin the wrong key on a VM's first connection.

Deploying a region

  1. .env: regional DOMAIN, cde_acme_* creds, ACME_EMAIL. No naming flag — the stack is regional by construction.
  2. docker compose up -d — certbot needs the cde.glueopshosted.com NS delegation live to validate (aws-dns-production#150).
  3. Smoke test: ssh -p 2222 -R smoketest:80:localhost:3000 <region>.tunnels.cde.glueopshosted.com → the printed URL must be https://smoketest.<region>... with no <user>- prefix.
  4. Continue with the region runbook in feat: regional tunnel endpoints for CDE tunnels and access URLs slackbot-developer-workspaces#499 (DNS/CloudFront via glueops-opentofu-workspaces/aws-cloud-development-environment-assets-production#74).

🤖 Generated with Claude Code

One stack, two roles: unset keeps the legacy central behavior
(append-user naming, argv-identical default), regional boxes set
--force-requested-subdomains so VMs bind their bare hostname
(<hostname>.<region>.tunnels.cde.glueopshosted.com). Documents the
role matrix, CloudFront origin relationship, and credential sources.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErBUiAYTosnpbF9hvUj3Dn
The stack now deploys only per-datacenter regional tunnel servers:
--force-requested-subdomains is hardcoded (bare-hostname binds, taken
names fail the bind), the SISH_SUBDOMAIN_FLAGS toggle is gone, and the
fork-migration runbook is removed. The legacy central box must stay
pinned to its current checkout (pre-0.4) — repo tip no longer carries
the append-user naming it depends on.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErBUiAYTosnpbF9hvUj3Dn
@venkatamutyala venkatamutyala changed the title feat: support regional tunnel deployments via SISH_SUBDOMAIN_FLAGS feat!: regional-only deployment — drop legacy append-user support Aug 2, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErBUiAYTosnpbF9hvUj3Dn
@github-actions github-actions Bot added the patch label Aug 2, 2026
venkatamutyala and others added 5 commits August 2, 2026 08:40
Supersedes renovate PRs #28/#29/#30 (taking 3.14 over 3.13). All pins
verified on the pinned digest: cp314 musllinux wheels present (MarkupSafe
C speedups active), Flask/Werkzeug CI-test 3.14 upstream; gunicorn 26.0.0
lacks official 3.14 metadata but the sync-worker path is exercised and
passing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErBUiAYTosnpbF9hvUj3Dn
Expert review of the TOFU authorizer found and reproduced: a non-string
auth_key left an empty key file behind, permanently locking the username
out; distinct raw usernames collapsing to one key file via
secure_filename ("vm 1" vs "vm_1"); and a create-then-write window
exposing empty files to concurrent workers. Registration now validates
JSON shape/types, rejects any username the sanitizer would alter, and
links a fully-written temp file into place atomically (same-key race
losers still allowed). Also: 16KB request cap, remote_addr in audit
lines, PYTHONUNBUFFERED so those lines actually flush.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErBUiAYTosnpbF9hvUj3Dn
Second expert pass (Python + sish v2.23.0 source review): cap usernames
at the 63-char hostname label limit (oversized attacker-chosen SSH
usernames reached the filesystem and turned every retry into a
traceback-spamming 500 from the public SSH port), deny on any OSError
reading a stored key instead of 500ing, and strip/reject whitespace-only
keys. Compose: authentication-key-request-timeout 15s -> 3s (sish holds
a global lock across the webhook call; a hung authenticator serialized
all auth on the instance) and point authentication-keys-directory at a
tmpfs so the webhook-bypass path is empty by construction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErBUiAYTosnpbF9hvUj3Dn
Each audit line is now a single write syscall (4 workers share stdout;
print()'s separate message+newline writes could interleave mid-line),
and a registration collision against an unreadable existing path (e.g.
a dangling symlink in sish_users/) audits "stored key unreadable"
instead of mislabeling it a race. Non-root container and a TOFU
registration cap were considered and deliberately skipped (compose
runs as root; bind-mount ownership would break registrations).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErBUiAYTosnpbF9hvUj3Dn
TUNNEL_ALLOWED_CIDRS (default: all private ranges) is enforced by the
authenticator using the client address sish reports on every SSH auth —
so it gates tunnel registration/connection only, and browsers/CloudFront
(:80/:443) are unaffected. Unparseable or missing sources fail closed
while the list is active; empty value disables the check; a malformed
CIDR crashes the authenticator at startup. sish's own --whitelisted-ips
was deliberately not used: it is global across SSH and HTTP/S and would
have blocked public web access.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErBUiAYTosnpbF9hvUj3Dn
@venkatamutyala
venkatamutyala force-pushed the feat/regional-tunnel-endpoints branch from 0aa78cb to 09fa290 Compare August 2, 2026 09:10
venkatamutyala and others added 2 commits August 2, 2026 09:14
100.64.0.0/10 joins the RFC 1918 defaults for TUNNEL_ALLOWED_CIDRS —
codespace VMs join the tailnet at boot, so tailnet-sourced tunnel
connections are first-class. Also drops the host-firewall documentation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErBUiAYTosnpbF9hvUj3Dn
sish v2.23.0 source review confirmed the allowlist gates every
tunnel-creation path — but an unqualified port publish also binds [::],
and docker-proxy relays IPv6 clients into the IPv4-only compose network
with their source rewritten to the bridge gateway, a 172.x address
inside the private-range allowlist. Binding 0.0.0.0 closes the
laundering path; tunnel DNS is A-record-only so no legitimate client
connects over v6.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErBUiAYTosnpbF9hvUj3Dn
@venkatamutyala
venkatamutyala merged commit 88f4713 into main Aug 2, 2026
2 checks passed
@venkatamutyala
venkatamutyala deleted the feat/regional-tunnel-endpoints branch August 2, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant