From 97347280140eb1dbc834f78dd78973a0025b435c Mon Sep 17 00:00:00 2001 From: Jonathan Murray Date: Fri, 22 May 2026 11:36:37 -0400 Subject: [PATCH 1/8] Add SSH troubleshooting guide Closes #180 --- docs/accessing-orcd/ssh-troubleshooting.md | 136 +++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 137 insertions(+) create mode 100644 docs/accessing-orcd/ssh-troubleshooting.md diff --git a/docs/accessing-orcd/ssh-troubleshooting.md b/docs/accessing-orcd/ssh-troubleshooting.md new file mode 100644 index 00000000..feb8d571 --- /dev/null +++ b/docs/accessing-orcd/ssh-troubleshooting.md @@ -0,0 +1,136 @@ +--- +tags: + - SSH + - Troubleshooting + - Logging In +--- + +# SSH Troubleshooting + +## Connection Refused or Timeout + +Make sure you are using the correct login command: +```bash +ssh [kerberos-username]@orcd-login.mit.edu +``` + +If the connection hangs or times out, you may be on a network that blocks SSH. Try a different network or contact [orcd-help@mit.edu](mailto:orcd-help@mit.edu). + +## Permission Denied + +**Check your username:** Make sure you are using your MIT Kerberos username, not your full email address. + +**Check your SSH key permissions:** Run these commands on your laptop: +```bash +chmod 700 ~/.ssh +chmod 600 ~/.ssh/id_rsa +``` + +**Key not uploaded:** If you have not yet set up SSH keys, see the [SSH Key Setup](ssh-setup.md) page. + +!!! note + Even with SSH keys set up, you will still be prompted for your MIT Kerberos password and Duo. This is expected. + +## Host Key Warning + +If you see a message like `WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED` or `This host key is known by the following other names`, run: +```bash +ssh-keygen -R orcd-login.mit.edu +``` +Then reconnect and type `yes` when prompted. This is common after system maintenance. + +!!! warning + If this happens repeatedly without any maintenance, contact [orcd-help@mit.edu](mailto:orcd-help@mit.edu). + +## Too Many Authentication Failures + +If you see: +``` +Received disconnect from ... : Too many authentication failures +``` + +Add the following to your `~/.ssh/config`: +``` +Host orcd-login.mit.edu + IdentityFile ~/.ssh/id_rsa + IdentitiesOnly yes +``` + +!!! note + This setting only applies to connections to `orcd-login.mit.edu` and will not affect SSH connections to any other servers. + +## Forgot SSH Key Passphrase + +There is no way to recover a forgotten passphrase. Generate a new SSH key pair and upload it to Engaging following the [SSH Key Setup](ssh-setup.md) instructions. + +## Dropped or Slow Connections + +Add the following to your `~/.ssh/config` to keep sessions alive: +``` +Host orcd-login.mit.edu + ServerAliveInterval 60 + ServerAliveCountMax 10 +``` + +For slow initial connections, also add `GSSAPIAuthentication no`: +``` +Host orcd-login.mit.edu + ServerAliveInterval 60 + ServerAliveCountMax 10 + GSSAPIAuthentication no +``` + +??? info "Other factors that can affect connection stability" + - **Wi-Fi instability** — a flaky wireless connection will drop SSH sessions regardless of keepalive settings. Try a wired connection if you experience frequent drops. + - **MIT VPN** — being connected to the MIT VPN can add latency depending on your network path. + - **Server-side idle timeout** — the server may enforce its own idle timeout. Client-side keepalives help but are not a guarantee against disconnection. + - **Large data transfers** — running a file transfer in the same session can saturate the connection and make the session appear hung. Use a separate terminal window for transfers. + +## Cannot Connect to a Compute Node + +Compute nodes are not directly accessible from outside the cluster, and you must have a running job on the node to connect to it. SSH to the login node first, then connect to the compute node: +```bash +ssh [kerberos-username]@orcd-login.mit.edu +ssh nodeXXXX +``` + +To connect directly from your laptop in one step using ProxyJump, see [Setting up your Config File](../recipes/vscode.md#setting-up-your-config-file) in the VS Code recipe for an example config. + +## IP Address Blocked + +If you have had repeated failed login attempts, our systems may have temporarily blocked your IP address. This will look like a connection timeout or `Connection refused` rather than an authentication error — your connection will not reach the server at all. + +If you suspect your IP address has been blocked, stop attempting to connect and wait 10 minutes. The block will lift automatically if no further failed attempts are made during that time. + +!!! warning + If you have VS Code open with a Remote SSH session, it may attempt to reconnect automatically in the background, which will reset the 10-minute timer and keep the block in place. Close VS Code before waiting for the block to clear. + +If the block does not clear after 10 minutes, open a ticket with [orcd-help@mit.edu](mailto:orcd-help@mit.edu) and include your IP address. You can find it with: + +=== "macOS/Linux" + ```bash + curl ifconfig.me + ``` +=== "Windows" + ``` + curl ifconfig.me + ``` + If that does not work, try: + ``` + (Invoke-WebRequest -Uri "ifconfig.me").Content + ``` + +## Account Locked + +10 failed Duo attempts will lock your account. This lock is managed by [MIT IS&T](https://ist.mit.edu/duo-security/duo), not ORCD, and will lift automatically after 90 minutes provided no further failed login attempts are made during that time. + +!!! warning + If you have VS Code open with a Remote SSH session, it may attempt to reconnect automatically in the background, resetting the 90-minute timer and keeping your account locked. Close VS Code before waiting for the lock to clear. + +To prevent this from happening again: + +- Log into [ORCD OnDemand](https://orcd-ood.mit.edu) — this satisfies Duo authentication and removes the Duo requirement for SSH for a few hours. +- Use [SSH Control Channels](control-channels.md) to reduce how often you need to authenticate with Duo. +- If you use VS Code, adjust the [VS Code Remote SSH settings](../recipes/vscode.md#adjust-the-remotessh-extension-settings). + +If your account does not unlock after 90 minutes, contact [orcd-help@mit.edu](mailto:orcd-help@mit.edu). diff --git a/mkdocs.yml b/mkdocs.yml index 2ea81fba..a76d2150 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -10,6 +10,7 @@ nav: - SSH Key Setup: accessing-orcd/ssh-setup.md - VSCode Remote SSH: recipes/vscode.md - SSH Control Channels: accessing-orcd/control-channels.md + - SSH Troubleshooting: accessing-orcd/ssh-troubleshooting.md - Filesystems and File Transfer: - General Use Filesystems: filesystems-file-transfer/filesystems.md # - Project Specific Filesystems: filesystems-file-transfer/project-filesystems.md From 93b6b074a1baae9e1e961278103a1debf0a1261c Mon Sep 17 00:00:00 2001 From: Jonathan Murray Date: Fri, 22 May 2026 11:48:34 -0400 Subject: [PATCH 2/8] Fix data transfer bullet in SSH troubleshooting guide --- docs/accessing-orcd/ssh-troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/accessing-orcd/ssh-troubleshooting.md b/docs/accessing-orcd/ssh-troubleshooting.md index feb8d571..061b0e41 100644 --- a/docs/accessing-orcd/ssh-troubleshooting.md +++ b/docs/accessing-orcd/ssh-troubleshooting.md @@ -84,7 +84,7 @@ Host orcd-login.mit.edu - **Wi-Fi instability** — a flaky wireless connection will drop SSH sessions regardless of keepalive settings. Try a wired connection if you experience frequent drops. - **MIT VPN** — being connected to the MIT VPN can add latency depending on your network path. - **Server-side idle timeout** — the server may enforce its own idle timeout. Client-side keepalives help but are not a guarantee against disconnection. - - **Large data transfers** — running a file transfer in the same session can saturate the connection and make the session appear hung. Use a separate terminal window for transfers. + - **Large data transfers** — data transfers to or from your local machine can saturate your network connection and make your SSH session appear slow or unresponsive. ## Cannot Connect to a Compute Node From fa0126c4b18669370bedb8d3f5f3e8e0def24203 Mon Sep 17 00:00:00 2001 From: Jonathan Murray Date: Fri, 22 May 2026 11:51:57 -0400 Subject: [PATCH 3/8] Remove email address reference from Permission Denied section --- docs/accessing-orcd/ssh-troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/accessing-orcd/ssh-troubleshooting.md b/docs/accessing-orcd/ssh-troubleshooting.md index 061b0e41..1a42ec77 100644 --- a/docs/accessing-orcd/ssh-troubleshooting.md +++ b/docs/accessing-orcd/ssh-troubleshooting.md @@ -18,7 +18,7 @@ If the connection hangs or times out, you may be on a network that blocks SSH. T ## Permission Denied -**Check your username:** Make sure you are using your MIT Kerberos username, not your full email address. +**Check your username:** Make sure you are using your MIT Kerberos username. **Check your SSH key permissions:** Run these commands on your laptop: ```bash From 41e6d8bc614b090c0219af9affd488913d923edb Mon Sep 17 00:00:00 2001 From: Jonathan Murray Date: Fri, 22 May 2026 13:37:13 -0400 Subject: [PATCH 4/8] Add ssh -vvv tip, move to bottom, standardize USERNAME placeholder --- docs/accessing-orcd/ssh-troubleshooting.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/accessing-orcd/ssh-troubleshooting.md b/docs/accessing-orcd/ssh-troubleshooting.md index 1a42ec77..83234c43 100644 --- a/docs/accessing-orcd/ssh-troubleshooting.md +++ b/docs/accessing-orcd/ssh-troubleshooting.md @@ -11,7 +11,7 @@ tags: Make sure you are using the correct login command: ```bash -ssh [kerberos-username]@orcd-login.mit.edu +ssh USERNAME@orcd-login.mit.edu ``` If the connection hangs or times out, you may be on a network that blocks SSH. Try a different network or contact [orcd-help@mit.edu](mailto:orcd-help@mit.edu). @@ -90,7 +90,7 @@ Host orcd-login.mit.edu Compute nodes are not directly accessible from outside the cluster, and you must have a running job on the node to connect to it. SSH to the login node first, then connect to the compute node: ```bash -ssh [kerberos-username]@orcd-login.mit.edu +ssh USERNAME@orcd-login.mit.edu ssh nodeXXXX ``` @@ -134,3 +134,11 @@ To prevent this from happening again: - If you use VS Code, adjust the [VS Code Remote SSH settings](../recipes/vscode.md#adjust-the-remotessh-extension-settings). If your account does not unlock after 90 minutes, contact [orcd-help@mit.edu](mailto:orcd-help@mit.edu). + +## Still Having Trouble? + +If you have worked through this guide and are still unable to connect, run the following command and include the output in your ticket to [orcd-help@mit.edu](mailto:orcd-help@mit.edu): +```bash +ssh -vvv USERNAME@orcd-login.mit.edu +``` +The `-vvv` flag enables verbose output and gives the support team detailed information about where the connection is failing. From 7198b977f1286f848b9317ce720c773f9204f55f Mon Sep 17 00:00:00 2001 From: Jonathan Murray Date: Fri, 22 May 2026 15:50:14 -0400 Subject: [PATCH 5/8] Add context explaining GSSAPIAuthentication no for slow connections --- docs/accessing-orcd/ssh-troubleshooting.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/accessing-orcd/ssh-troubleshooting.md b/docs/accessing-orcd/ssh-troubleshooting.md index 83234c43..7eb49203 100644 --- a/docs/accessing-orcd/ssh-troubleshooting.md +++ b/docs/accessing-orcd/ssh-troubleshooting.md @@ -72,7 +72,7 @@ Host orcd-login.mit.edu ServerAliveCountMax 10 ``` -For slow initial connections, also add `GSSAPIAuthentication no`: +If your connection is slow to establish — particularly when connecting from off-campus — also add `GSSAPIAuthentication no`: ``` Host orcd-login.mit.edu ServerAliveInterval 60 @@ -80,6 +80,8 @@ Host orcd-login.mit.edu GSSAPIAuthentication no ``` +By default, SSH attempts Kerberos (GSSAPI) authentication before trying your password or SSH key. This requires a handshake with MIT's Kerberos infrastructure, which can add several seconds to your connection time if you are on a slow or off-campus network. Disabling it tells SSH to skip straight to password and key authentication. This has no effect on users who do not use Kerberos tickets to authenticate. + ??? info "Other factors that can affect connection stability" - **Wi-Fi instability** — a flaky wireless connection will drop SSH sessions regardless of keepalive settings. Try a wired connection if you experience frequent drops. - **MIT VPN** — being connected to the MIT VPN can add latency depending on your network path. From 805b8e417d66debfb826e7a37fe1ab442a815bbb Mon Sep 17 00:00:00 2001 From: Jonathan Murray Date: Wed, 10 Jun 2026 14:50:58 -0400 Subject: [PATCH 6/8] Update docs/accessing-orcd/ssh-troubleshooting.md Co-authored-by: Lauren Milechin <13963460+lmilechin@users.noreply.github.com> --- docs/accessing-orcd/ssh-troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/accessing-orcd/ssh-troubleshooting.md b/docs/accessing-orcd/ssh-troubleshooting.md index 7eb49203..884c24c4 100644 --- a/docs/accessing-orcd/ssh-troubleshooting.md +++ b/docs/accessing-orcd/ssh-troubleshooting.md @@ -37,7 +37,7 @@ If you see a message like `WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED` or ` ```bash ssh-keygen -R orcd-login.mit.edu ``` -Then reconnect and type `yes` when prompted. This is common after system maintenance. +Then reconnect and type `yes` when prompted. This is sometimes happens after system maintenance. !!! warning If this happens repeatedly without any maintenance, contact [orcd-help@mit.edu](mailto:orcd-help@mit.edu). From 841ddbcf061898eeeeab4d494da5cf226a2e1783 Mon Sep 17 00:00:00 2001 From: Jonathan Murray Date: Wed, 10 Jun 2026 14:51:16 -0400 Subject: [PATCH 7/8] Update docs/accessing-orcd/ssh-troubleshooting.md Co-authored-by: Lauren Milechin <13963460+lmilechin@users.noreply.github.com> --- docs/accessing-orcd/ssh-troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/accessing-orcd/ssh-troubleshooting.md b/docs/accessing-orcd/ssh-troubleshooting.md index 884c24c4..44623000 100644 --- a/docs/accessing-orcd/ssh-troubleshooting.md +++ b/docs/accessing-orcd/ssh-troubleshooting.md @@ -14,7 +14,7 @@ Make sure you are using the correct login command: ssh USERNAME@orcd-login.mit.edu ``` -If the connection hangs or times out, you may be on a network that blocks SSH. Try a different network or contact [orcd-help@mit.edu](mailto:orcd-help@mit.edu). +If the connection hangs or times out, you may be on a network that blocks SSH, or your IP address has been blocked (see [below](#ip-address-blocked)). Try a different network or contact [orcd-help@mit.edu](mailto:orcd-help@mit.edu). ## Permission Denied From a5e0ab90d459927113a1aa58a9ca72d9b391ce31 Mon Sep 17 00:00:00 2001 From: Jonathan Murray Date: Wed, 10 Jun 2026 14:51:29 -0400 Subject: [PATCH 8/8] Update docs/accessing-orcd/ssh-troubleshooting.md Co-authored-by: Lauren Milechin <13963460+lmilechin@users.noreply.github.com> --- docs/accessing-orcd/ssh-troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/accessing-orcd/ssh-troubleshooting.md b/docs/accessing-orcd/ssh-troubleshooting.md index 44623000..50578129 100644 --- a/docs/accessing-orcd/ssh-troubleshooting.md +++ b/docs/accessing-orcd/ssh-troubleshooting.md @@ -133,7 +133,7 @@ To prevent this from happening again: - Log into [ORCD OnDemand](https://orcd-ood.mit.edu) — this satisfies Duo authentication and removes the Duo requirement for SSH for a few hours. - Use [SSH Control Channels](control-channels.md) to reduce how often you need to authenticate with Duo. -- If you use VS Code, adjust the [VS Code Remote SSH settings](../recipes/vscode.md#adjust-the-remotessh-extension-settings). +- If you use VS Code, adjust the [VS Code Remote SSH settings](../recipes/vscode.md#adjust-the-remotessh-extension-settings) to limit reconnection attempts. If your account does not unlock after 90 minutes, contact [orcd-help@mit.edu](mailto:orcd-help@mit.edu).