Skip to content

fix(ssh): suppress cascading DNS error when sign mode fails - #2430

Draft
brollb wants to merge 1 commit into
mainfrom
brollb/fix-ssh-errors
Draft

fix(ssh): suppress cascading DNS error when sign mode fails#2430
brollb wants to merge 1 commit into
mainfrom
brollb/fix-ssh-errors

Conversation

@brollb

@brollb brollb commented May 7, 2026

Copy link
Copy Markdown
Contributor

🚀 What

When SSHing into a training job that isn't in RUNNING state, users were seeing two error messages:

  1. baseten-ssh: API error (400): Training job must be in RUNNING state ... ← the relevant one
  2. ssh: Could not resolve hostname ...: nodename nor servname provided, or not known ← confusing noise

This PR removes the second spurious error.

💻 How

The SSH config uses a Match exec block that runs proxy-command.py --sign before the connection. SSH only applies the Match block (and thus ProxyCommand) if the exec command exits 0. Previously, any sign failure called error()sys.exit(1), causing SSH to skip ProxyCommand and attempt direct DNS resolution of the *.ssh.baseten.co hostname — which fails with the confusing secondary error.

Fix:

  • main_sign() now always exits 0. In sign mode, error() suppresses stderr output and saves the message to a per-hostname sign-error cache file (~/.ssh/baseten/.jwt-cache/<hostname>.sign-error).
  • main_proxy() checks the cache at startup. If a sign error is present, it clears the file, prints the message, and exits 1 — so users see exactly one, actionable error message.

🔬 Testing

Manually verified the two-error scenario is reduced to one. Existing unit tests in truss/tests/cli/test_ssh.py all pass (43/43).

When the SSH sign step fails (e.g. job not in RUNNING state), the proxy
command was exiting with code 1. This caused SSH to skip the Match block
entirely, so instead of running ProxyCommand it tried to resolve the
hostname via DNS, producing a confusing secondary error:
  "Could not resolve hostname ... nodename nor servname provided"

Fix by making main_sign() always exit 0:
- error() in sign mode suppresses stderr output and saves the message
  to a per-hostname sign-error cache file
- main_proxy() checks that cache first and surfaces the error cleanly,
  so users see only the relevant message (e.g. API 400 state error)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Brian Broll seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants