feat(train): print SSH commands after push, add truss ssh --training-job-id - #2446
Open
rcano-baseten wants to merge 1 commit into
Open
feat(train): print SSH commands after push, add truss ssh --training-job-id#2446rcano-baseten wants to merge 1 commit into
truss ssh --training-job-id#2446rcano-baseten wants to merge 1 commit into
Conversation
…-job-id` After `truss train push`, users had to know the SSH hostname format and manually poll status to know when SSH was reachable. This adds: - Per-node `ssh training-job-<id>-<n>.ssh.baseten.co` lines in the push success output, plus a hint about `truss ssh setup`. - `truss ssh --training-job-id <id> [--node-id 0]` that waits for the job to be running (reusing TrainingPollerMixin, the same primitive `truss train logs --tail` uses) and then `os.execvp`s ssh into the selected node. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cretz
reviewed
May 11, 2026
| ) | ||
| @click.option("--remote", type=str, required=False, help="Remote to use.") | ||
| @click.pass_context | ||
| def ssh( |
Contributor
There was a problem hiding this comment.
I think this should work for inference too, though in general we recommend users use their own SSH clients configured how they want, with the args they want, etc. Is this meant to be our newly suggested way to SSH into training/inference containers, or is it just a helper?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 What
After
truss train push, users currently have to know the SSH hostname format (training-job-<job_id>-<node>.ssh.baseten.co) and manually polltruss train view/truss train logsto know when SSH is reachable.This PR adds two things:
_print_training_job_success_messagenow appendsssh training-job-<id>-<n>.ssh.baseten.cofor each node (labels node 0(leader)for multi-node), plus a hint abouttruss ssh setup.truss ssh --training-job-id <id> [--node-id 0]. Waits for the job to be running, thenexecssshso you land directly in the shell. No-A.💻 How
truss/cli/train_commands.py: small_format_ssh_commands(job_id, node_count)helper, and added an SSH section to the existing success message. Format mirrors the one already used intruss train workstation.truss/cli/ssh_commands.py: converted thesshClick group toinvoke_without_command=Trueso it accepts options directly. Flow:is_setup_complete()(already intruss/cli/ssh.py); errors with atruss ssh setuphint if missing.project_idvia the existingget_most_recent_job(remote_provider, None, job_id)— same helpertruss train logsuses.node_countfrom the job'sinstance_typeand reject out-of-range--node-id.TrainingPollerMixin.before_polling()for the wait loop — same primitivetruss train logs --tailuses. Spinner messages already handle "Waiting for GPU capacity (job is queued)…" / "Waiting for job to deploy…".TRAINING_JOB_RUNNING,os.execvp("ssh", ["ssh", "training-job-<id>-<n>.ssh.baseten.co"]).Out of scope
--deployment-idfor inference models — needsmodel_idupfront, no clean reverse lookup today. Will be a follow-up.🔬 Testing
truss/tests/cli/test_ssh.py:_format_ssh_commandsfor single-node and multi-node.truss ssh --training-job-id: correct hostname, default node-id is 0, errors when SSH setup is incomplete, errors on out-of-range--node-id, errors when job is not running.uv run ruff checkclean on touched files.TestSetupSSHConfigfailures onmain(env can't findpython3.xviashutil.which) are unrelated.End-to-end manual verification (to do against a real remote):
truss train push <config>prints the SSH lines (1 line for single-node, 2+ for multi-node).truss ssh --training-job-id <id>against aQUEUEDjob shows the spinner and updates as state changes.RUNNING, lands in the shell on the right node.--node-id 9(out of range) gives a clean error.truss ssh setup, gives the setup hint.🤖 Generated with Claude Code