I intended to post in defense of manpages. I love manpages. I think most open-source manpages are high-quality, and a few are really outstanding.
"It's easier to ask an AI" can be true without implying that manpages are bad.
However, "man" the tool does have issues, and one of them bit me just now.
So, I didn't know about openssh client escapes like ~?. I thought, "surely that's in the manpage?" I opened the manpage (in less) and searched for "\~\?". No hits.
Of course, escape characters are documented in the manpage, and the string "~?" does appear. Why didn't search find it? Because man, in its infinite wisdom, decided to render every instance of "~" as some bizarre unicode not-tilde, which is visually similar but totally impossible to grep for.
This has also bitten me in the past with dash. DASH. A character that is critically important when documenting invocation options. man loves to convert it into something that looks like dash, prints like dash, but doesn't come up in search.
I'm sure there is a way to turn this "feature" off, and I'm about to spend a bunch of time figuring out what it is. But this is documentation for command-line tools. Silently destroying our ability to grep it should NOT be the default.
Manpages are both good and awful. If you already know how to use the command and you just need a refresher on how to enable the diagnostic option for wolves, they're fine. OTOH if you don't know how to use the command they're terrible, just an endless catalogue of incomprehensible options scrolling past. Problem is that that's what Dennis and Ken created in 1929 for use on ASR-33s and it hasn't been updated since. 99% of the time when I need to use some new unfamiliar command I go to whatever DDG pops up, usually StackOverflow, for a guide on usage.
I don't know about it changing your characters, but you can search for a string verbatim by pressing ctrl-r at the start of the search pattern, like this:
yeah it sounds like your environment is doing some weird mangling of the encoding. Maybe consider adding an 'alias man="man -E ascii"' into your .bashrc
You might also consider seeing if it's any different in WSL. Or you could, you know, just use Linux ;P
I've been using the ~. shortcut for a while, but somehow escaped learning about the help menu.
Another neat thing I noticed while playing with it just now: there's an option to enter ~ twice to send a literal ~, but usually you don't have to do this when typing something like 'ls ~' in a regular session. Not only does the ~ have to be the first character on a line to start an escape sequence, but typing on a line, backspacing all the way to the start and then typing ~ also sends a literal tilde. It only triggers the escape sequence if the ~ is the chronologically first character after a newline (or first in the session), which is an unlikely thing to type into a shell in a normal session. Good choice of UI, both the character and the state machine.
> typing on a line, backspacing all the way to the start and then typing ~ also sends a literal tilde
for the younger readers, yes, because in terminal echo mode, "backspacing" does not clear your terminal line buffer, those characters backspaced are already sent on the line. if you ever seen a misconfigured terminal, it hints what's going on, like:
user@host$ ls ~/^?^?^?^?^?~/a.out
^? is backspace's control char.
that is ssh watches what you type, not what is on the screen (terminal).
The drawback is that if you think your session is hanging and want to bail with ~., you have to press enter, which might actually make it to the server and execute something.
I've been using SSH for ~15 years and never knew about these escape sequences. I'm eagerly awaiting my next hung session so that I can test `~.`. It's much nicer than my current approach of having to close that terminal window.
If hung SSH connections are common it's likely due to CGNAT which use aggressively low TCP timeouts. e.g. I've found all UK mobile carriers set their TCP timeout as low as 5 minutes. The "default" is supposed to be 2 hours, you could literally sleep your computer, zero packets, and an SSH connection would continue to work an hour later, and generally speaking this is still true unless CGNAT is in the way.
If you are interested there are a few ways you can fix this:
Easiest is to use a VPN, because the VPN's exit node becomes the effective NAT they usually have normal TCP timeouts due to being less resource constrained. Another nice benefit of this method is you can move between physical networks and your connection doesn't die... If you use Tailscale then you already have this in a more direct way.
Another is to tune the tcp_keepalive kernel parameters. Lowering the keepalive timeout to be less than the CGNAT timeout will cause keepalive probes to prevent CGNAT from dropping the connection even while your SSH connection is technically idle. For Linux I pop these into /etc/sysctl.d/z.conf, I have no idea for Windows or Mac:
This is really a misuse of these settings, they are supposed to be for checking TCP connections are still alive and clearing them up from the local routing table. Instead the idea is to exploit the probes by sending them more frequently to force idle connections to stay alive in a CGNAT environment (dont worry the probes are tiny and still very infrequent).
_time=240 will send a probe after 4 mins of idle connection instead of the default 2 hours, undercutting the CGNAT timeout. _intvl=60 and _probes=120 mean it will send 120 probes 60 seconds apart (2 hours worth) before considering the connection dead. This will keep it alive for at least 2 hours, but also allows us to have the best of both worlds so that under a nice NAT it keeps the old behaviour, e.g if I temporarily lose my network the SSH connection is still valid after 2 hours, but under CGNAT it will at least not drop the connection after 5 mins so long as I keep my computer on and don't lose the network.
There are also some SSH client keepalive settings but I'm less familiar with them.
Check Mosh. It supports these kind of cuts and it will reconnect seamlessly. It will use far less bandwidth too.
I successfully tried it with a 2.7 KBPS connection.
Depends on whether your sockets survive that, though. Especially on Wi-Fi, many implementations will reset your interface when sleeping, and sockets usually don't survive that.
Even if they do, if the remote side has heartbeats/keepalive enabled (at the TCP or SSH level), your connection might be torn down from the server side.
Yes, by generally I really mean all the defaults are pretty permissive, but I understand some people tune both TCP and SSH on their servers to drop connections faster because they are worried about resource exhaustion.
But if you throw up a default Linux install for your SSH box and have a not-horrible wifi router with a not-horrible internet provider then IME you can sleep your machine and keep an SSH connection alive for quite some time... I appreciate that might be too many "not-horrible" requirements for the real world today though.
Yes, this makes your connection more likely not survive client suspends. (ClientAliveInterval, which makes the server ping the client, will make it fail almost certainly, since the server will be active while the client is sleeping.)
Well, for different reasons, but you have similar issues with IPv6 as well. If your client uses temporary addresses (most likely since they're enabled by default on most OS), OpenSSH will pick one of them over the stable address and when they're rotated the connection breaks.
For some reason, OpenSSH devs refuse to fix this issue, so I have to patch it myself:
--- a/sshconnect.c
+++ b/sshconnect.c
@@ -26,6 +26,7 @@
#include <net/if.h>
#include <netinet/in.h>
#include <arpa/inet.h>
+#include <linux/ipv6.h>
#include <ctype.h>
#include <errno.h>
@@ -370,6 +371,11 @@ ssh_create_socket(struct addrinfo *ai)
if (options.ip_qos_interactive != INT_MAX)
set_sock_tos(sock, options.ip_qos_interactive);
+ if (ai->ai_family == AF_INET6 && options.bind_address == NULL) {
+ int val = IPV6_PREFER_SRC_PUBLIC;
+ setsockopt(sock, IPPROTO_IPV6, IPV6_ADDR_PREFERENCES, &val, sizeof(val));
+ }
+
/* Bind the socket to an alternative local IP address */
if (options.bind_address == NULL && options.bind_interface == NULL)
return sock;
I'm not sure what happens to the socket, maybe it's closed and reopened, but with this patch I have SSH sessions lasting for days with no issues. Without it, even roaming between two access points can break the session.
It would also seem to break address privacy (usually not much of a concern if you authenticate yourself via SSH anyway, but still, it leaks your Ethernet or Wi-Fi interface's MAC address in many older setups).
Not anonymous, but it's pretty unexpected for different servers with potentially different identities for each to learn your MAC address (if you're using the default EUI-64 method for SLAAC).
This is a very common misconception. The issue is not IPv4 or CGNAT, it's stateful middleboxes... of which IPv6 has plenty.
The largest IPv6 deployments in the world are mobile carriers, which are full of stateful firewalls, DPI, and mid-path translation. The difference is that when connections drop it gets blamed on the wireless rather than the network infrastructure.
Also, fun fact: net.ipv4.tcp_keepalive_* applies to IPv6 too. The "ipv4" is just a naming artifact.
Mobile carriers usually have stateful firewalls for IPv6 as well (otherwise you can get a lot of random noise on the air interface, draining both your battery and data plan), so it's an issue just the same.
The constrained resource there is only firewall-side memory, though, as opposed to that plus (IP, port) tuples for CG-NAT.
Or my predecessor/address space neighbor, or that of somebody using my wireless hotspot once, or that of me clicking a random link once and connecting to 671 affiliated advertisers's analytics servers...
I think a default policy of "no inbound connections" does makes sense for most mobile users. It should obviously be configurable.
Have been using that weekly since probably 20 years. Will change your life :)
My other favourite is I very often SSH with -v to figure out why the connection is hanging, you rapidly figure out if DNS is failing, the TCP connection doesn't open, it does open but no traffic flows at all or it opens and SSH negotiation starts but never finishes. You can learn a lot just from this about what is wrong.
And of course, you can use the ~v / ~V commands (as listed in the ~? menu) to increase/decrease verbosity after the connection is established.
That lets you `ssh -vvvv` to a host then once you've figured out the issue use ~V to decrease verbosity so that debug messages don't clutter your shell.
Also helps with auth failures, I've used it several times with co-workers who can't figure out why their ssh key isn't working. It lists the keys out and some extra information.
> It's much nicer than my current approach of having to close that terminal window.
You can also just kill the ssh process (say from another terminal). That way you get to keep your terminal window. And this works with everything "blocking" your terminal, not just ssh.
I last used this menu about 20 years ago when a dialup modem was the only way to roll, and have pretty much forgotten about it since the days of always-on direct to the desktop TCP/IP ..
Does not for me, not even with busybox sh and no funky escape codes in PS1 at all. It does with cat or yes running, so just something being output is not the problem… Hm.
Same with me, I'll still instinctively go for ~. when a connection has hung / dropped (usually because of a NAT via a rebooted firewall), but never even considered how ~ doesn't normally cause an issue. Never knew it had to be immediately following a newline. Also never knew about the other options, ~^Z in particular looks useful.
I wonder if anyone still remembers the ctrl-[ sequence in telnet. I think I only ever used the quit command in that though.
No need – your local ssh client, which interprets the escape sequence, doesn't have any understanding of whatever mode the remote session might currently be in.
The point of the return is to prime it to accept the start of a new escape sequence. Presumably the idea is that `~.` is not completely unlikely to occur as part of text entered remotely, but less so at the beginning of a new line.
I typically just create a "new" connection in a separate tab when I want to add tunneling.
I put new in quotes because I use another little-known feature, "ControlMaster". Multiplexes multiple connections into one, it makes making " new" sessions instant (can also be configured to persist a bit after disconnecting). Also useful for tab-completing remote paths. It does not prompt for authentication again, though. And it's a bit annoying when the connection hands (can be solved with ssh -o close, IIRC).
> I use another little-known feature, "ControlMaster". Multiplexes multiple connections into one, it makes making " new" sessions instant
Is this what secureCRT used as well? I remember this being all the rage back when I used windows, and it allowed this spawn new session by reusing the main one.
I'm using that as well but had issues with tunneling where it creates the tunnel in the background and terminates and so you might not know the random port it assigned or I couldn't figure out how to un-tunnel it and tunnel again to the same port. Just bypassed the control master then.
You think that one is cool, go check out "~?" in IPMI "sol activate". From there you can deliver a serial break to the kernel, which then gets you to a third rarely seen menu from the kernel's console, which allows you to do kernel debugging of various sorts.
I use it when I need to crash a kernel on purpose to test kdump over the network.
You can also send commands to the simulated console of a VM under libvirt with "virsh connect". But I don't think you can send a break to the kernel with that.
"secret" not in the sense that it's hidden, but that most people won't know about it. Because approximately nobody actually reads man pages in their entirety, they just get in to find out how a specific flag works and then get out.
the difference is that knowing 2^8 is generally not useful to people who don't know it
this here is something that's pretty useful to most ssh users, yet seldom spoken of
a better analogy would be comparing it to calling a very good, but not well-known restaurant a secret place - using the word to mean a hidden gem rather than an intentionally hidden secret
The ~ (tilde) escape character in UNIX cu (call UNIX) did the same thing in the 1970s, when the remote system was on the other side of a telephone modem or other serial port. I think it's safe to say that ssh is a descendant of cu. (I don't mean that cu printed a help menu, I mean it had the tilde escape.)
The ControlMaster mention deserves a full config block since the defaults make it awkward:
Host *
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h:%p
ControlPersist 10m
`mkdir -p ~/.ssh/sockets` first. With this setup, every subsequent ssh/scp/rsync to the same host reuses the existing master connection — no re-auth, near-instant open. ControlPersist keeps the socket alive for 10 minutes after the last session closes, so short gaps don't force a new handshake.
This pairs well with the ~C escape discussed here: adding a port forward mid-session via ~C -L 8080:localhost:8080 doesn't require a new connection or re-authentication when a ControlMaster is already running. Useful for those "I need to tunnel something I didn't anticipate" moments.
it starts with a pretty common char, but almost never gets in the way to the point I forget it exists. Meanwhile docker -t uses ^P which I use all the time for history instead of arrow keys. It's possible to configure it, but it's not worth the hassle on servers. Really, really annoying.
This is not specific to ssh. Telnet and rlogin have similar things with ~ as the escape character. Back in the day it was common to send BREAK and other escape sequences when you were hardwired.
Wow, never knew this... That said I'm not sure if I'll remember using it as my muscle memory is already trained to kill hung processes via ctrl-b s, ctrl-k, ctrl-b x (in tmux).
Unlike CTRL ], at least ~. doesn't require that I press two modifiers at the same time ... CTRL ALTGR $. Because people who define those kinds of shortcuts never consider how they might work on non-QWERTY layouts.
Don't tell people this. In a minute you'll realize that this menu can only mean that ssh runs multiple channels, and so you can start up other things over an established ssh connection.
Then the firewall guys will realize that they really can't allow ssh ... let's just not go there. Working at a bank is annoying enough already.
When your firewall guys "block" ssh (or anything else), all you need to do is run your ssh server on port 443 and then `ssh -p 443 user@host`.
Running on 443 means it will blend in with https traffic, making it much more difficult for them to detect and block (I've never actually seen anywhere that can detect/block it - you'd need to be doing deep packet inspection on encrypted traffic) :)
Not on the same IP at the same time, but you could always add a second IP for ssh. Just configure sshd to only listen on one IP and <webserver> to listen on the other.
I have noticed it while running ~/bin/some_command. The ~ doesn't echo until I also type the /. It doesn't cause any misbehavior because there is no binding for ~/ but can be slightly surprising.
I find it odd that you would have commands in ~/bin but not have it be the highest priority in your PATH. I use ~/.local/bin, but would never type it because i wouldn't have bins that overlap shell commands and no other path would have priority.
Usually, it is. IIRC, this was when I was just setting up my environment on a new host, after I had populated ~/bin but before I restarted my shell to pick up PATH modifications.
I'd guess this is because it only works in ssh PTY sessions. So it would have no effect on tunneling or when piping arbitrary data through ssh to a non-interactive remote command (unless you use the -t switch to force PTY allocation even when stdin is not a TTY).
I was never able to properly parse large man pages, I'm so happy that llms can now prepare half a usable command without spending an hour reading a time without a single usage example.
I've also been running (neo)vim as a manpager. You get the same features as with vim (like easily copying text or opening referenced files/other manpages without using the mouse), but neovim also parses the page and creates a table of contents, which can be used for navigation within the page. It doesn't always work perfectly, but is usually better than nothing.
You can also run commands when a user authenticates, grab their keys from github.com/username.keys, validate they're a user in a specific github group, then let them connect by outputing the keys, otherwise nothing to deny them access.
It's really great for ops teams where you want to give ssh access and manage it from github teams without needing a complex system.
Honest question, why is ProxyCommand `fun`? What do I get out of ProxyCommand that i do not get out of setting the correct order for ProxyJump and doing an ssh finalhost -- domy --bidding?
ProxyJump is a newer functionality. There used to be only ProxyCommand. ProxyJump is a shortcut for the usual way to use ProxyCommand to connect through a bastion host but ProxyCommand is more flexible. For example with ProxyCommand you can run any command to connect to the remote host. ProxyJump only connects over ssh. I think I replaced all my ProxyCommand with ProxyJump because I don't need much else than the normal use case.
You can get a lot more out of ProxyCommand. For example, you can run SSH over non-IP protocols, such as serial, Bluetooth RFCOMM for embedded boards, or vsock for virtual machines without networking set up at all. The latter is built into and setup up automatically by systemd:
ProxyCommand allows you to use any command to setup a connection. Not necessarily an ssh command, like ProxyJump. It can be any command, as long as it receives on stdin and produces on stdout, it can act like a TCP connection.
ProxyJump is a special case of `ProxyCommand ssh -p <port> <user>@<host>`. Can't replace the `ssh` in there when using ProxyJump.
I came across ProxyCommand earlier this week, funnily enough. I have Cloudflare Zero Trust set up with an SSH service[0], and have the server firewall drop all incoming traffic. That helps reduce my attack surface, since I don't have any incoming ports open.
I use ProxyCommand in edge-case devices where key auth is not an option and the password is not controlled by me. ProxyCommand points to a script the retrieves the password from the vault, puts it on the clipboard for pasting, reminds me via stderr it's done so, and then proxies the connection.
Interesting. I might have such a use case. Do you have anything about best practices on how to automate grabbing passwords from vaults? Cuz it seems to me that the vault needs to be kept open or keep the vault password somewhere on disk.
`pgrep`/`pkill` HATE this trick! Learn how this renegade developer (relatively) easily exits their hanging SSH sessions without restarting their laptop.
I still struggle with the fact that I was (or delusional) an effective C programmer "back in the day" (before google etc) and all we had was "man" to look up std/x11 system calls.
Now I am dismayed with juniors who can't even be bothered to use google (or llms) to look up stuff on their first hiccup.
If the ssh man page were any lazier it would just be a list of arguments with no context at all, so I get the frustation. The tricky bit is that much of the "secret menu" knowledge about ssh only lives in blog posts and random issue comments, which makes the doc gap feel intentional at times. If you want real examples or usage patterns you pretty much have to scrape Stack Overflow or crawl through dotfiles in public repos hoping someone else already fought the same fight.
Hidden or undocumented features like this always have a strange appeal. Part of it is nostalgia for older software where small Easter eggs or experimental features would sometimes ship in production builds.
I intended to post in defense of manpages. I love manpages. I think most open-source manpages are high-quality, and a few are really outstanding.
"It's easier to ask an AI" can be true without implying that manpages are bad.
However, "man" the tool does have issues, and one of them bit me just now.
So, I didn't know about openssh client escapes like ~?. I thought, "surely that's in the manpage?" I opened the manpage (in less) and searched for "\~\?". No hits.
Of course, escape characters are documented in the manpage, and the string "~?" does appear. Why didn't search find it? Because man, in its infinite wisdom, decided to render every instance of "~" as some bizarre unicode not-tilde, which is visually similar but totally impossible to grep for.
This has also bitten me in the past with dash. DASH. A character that is critically important when documenting invocation options. man loves to convert it into something that looks like dash, prints like dash, but doesn't come up in search.
I'm sure there is a way to turn this "feature" off, and I'm about to spend a bunch of time figuring out what it is. But this is documentation for command-line tools. Silently destroying our ability to grep it should NOT be the default.
Manpages are both good and awful. If you already know how to use the command and you just need a refresher on how to enable the diagnostic option for wolves, they're fine. OTOH if you don't know how to use the command they're terrible, just an endless catalogue of incomprehensible options scrolling past. Problem is that that's what Dennis and Ken created in 1929 for use on ASR-33s and it hasn't been updated since. 99% of the time when I need to use some new unfamiliar command I go to whatever DDG pops up, usually StackOverflow, for a guide on usage.
I don't know about it changing your characters, but you can search for a string verbatim by pressing ctrl-r at the start of the search pattern, like this:
Works for me on Debian (with the default pager, less).
There was more, then less, but once upon a time there was 'most'.
I miss most.
Thanks for the suggestions. Invoking man with "-E ascii" fixed this for me.
It sounds like some distributions do have this fixed in their default settings. I'm on Cygwin 3.6.6; maybe it's mostly a Cygwin thing.
Cygwin is nice, but it's a Frankenstein's monster. I'd hold back critical thoughts and feelings, until testing elsewhere.
You'll be happiest.
yeah it sounds like your environment is doing some weird mangling of the encoding. Maybe consider adding an 'alias man="man -E ascii"' into your .bashrc
You might also consider seeing if it's any different in WSL. Or you could, you know, just use Linux ;P
with neovim as the pager for man, it does find ~?
though you have to be aware of the escapes for regex, so \~?
\~\? works on macOS, as well as CentOS.
I've been using the ~. shortcut for a while, but somehow escaped learning about the help menu.
Another neat thing I noticed while playing with it just now: there's an option to enter ~ twice to send a literal ~, but usually you don't have to do this when typing something like 'ls ~' in a regular session. Not only does the ~ have to be the first character on a line to start an escape sequence, but typing on a line, backspacing all the way to the start and then typing ~ also sends a literal tilde. It only triggers the escape sequence if the ~ is the chronologically first character after a newline (or first in the session), which is an unlikely thing to type into a shell in a normal session. Good choice of UI, both the character and the state machine.
> typing on a line, backspacing all the way to the start and then typing ~ also sends a literal tilde
for the younger readers, yes, because in terminal echo mode, "backspacing" does not clear your terminal line buffer, those characters backspaced are already sent on the line. if you ever seen a misconfigured terminal, it hints what's going on, like:
user@host$ ls ~/^?^?^?^?^?~/a.out
^? is backspace's control char.
that is ssh watches what you type, not what is on the screen (terminal).
Sometimes it’s ^H. You can use `stty -a` to inspect the current setting (backspace is what’s called “erase” there).
The drawback is that if you think your session is hanging and want to bail with ~., you have to press enter, which might actually make it to the server and execute something.
many get used to Ctrl-U, Return, ~, period keystroke sequence for this.
For those of us in today's 10000, Ctrl-U is the default readline shortcut for unix-line-discard (see https://www.gnu.org/software/bash/manual/html_node/Commands-... and https://susam.net/unix-line-discard.html).
I've been using SSH for ~15 years and never knew about these escape sequences. I'm eagerly awaiting my next hung session so that I can test `~.`. It's much nicer than my current approach of having to close that terminal window.
If hung SSH connections are common it's likely due to CGNAT which use aggressively low TCP timeouts. e.g. I've found all UK mobile carriers set their TCP timeout as low as 5 minutes. The "default" is supposed to be 2 hours, you could literally sleep your computer, zero packets, and an SSH connection would continue to work an hour later, and generally speaking this is still true unless CGNAT is in the way.
If you are interested there are a few ways you can fix this:
Easiest is to use a VPN, because the VPN's exit node becomes the effective NAT they usually have normal TCP timeouts due to being less resource constrained. Another nice benefit of this method is you can move between physical networks and your connection doesn't die... If you use Tailscale then you already have this in a more direct way.
Another is to tune the tcp_keepalive kernel parameters. Lowering the keepalive timeout to be less than the CGNAT timeout will cause keepalive probes to prevent CGNAT from dropping the connection even while your SSH connection is technically idle. For Linux I pop these into /etc/sysctl.d/z.conf, I have no idea for Windows or Mac:
This is really a misuse of these settings, they are supposed to be for checking TCP connections are still alive and clearing them up from the local routing table. Instead the idea is to exploit the probes by sending them more frequently to force idle connections to stay alive in a CGNAT environment (dont worry the probes are tiny and still very infrequent)._time=240 will send a probe after 4 mins of idle connection instead of the default 2 hours, undercutting the CGNAT timeout. _intvl=60 and _probes=120 mean it will send 120 probes 60 seconds apart (2 hours worth) before considering the connection dead. This will keep it alive for at least 2 hours, but also allows us to have the best of both worlds so that under a nice NAT it keeps the old behaviour, e.g if I temporarily lose my network the SSH connection is still valid after 2 hours, but under CGNAT it will at least not drop the connection after 5 mins so long as I keep my computer on and don't lose the network.
There are also some SSH client keepalive settings but I'm less familiar with them.
Check Mosh. It supports these kind of cuts and it will reconnect seamlessly. It will use far less bandwidth too. I successfully tried it with a 2.7 KBPS connection.
> you could literally sleep your computer,
Depends on whether your sockets survive that, though. Especially on Wi-Fi, many implementations will reset your interface when sleeping, and sockets usually don't survive that.
Even if they do, if the remote side has heartbeats/keepalive enabled (at the TCP or SSH level), your connection might be torn down from the server side.
Yes, by generally I really mean all the defaults are pretty permissive, but I understand some people tune both TCP and SSH on their servers to drop connections faster because they are worried about resource exhaustion.
But if you throw up a default Linux install for your SSH box and have a not-horrible wifi router with a not-horrible internet provider then IME you can sleep your machine and keep an SSH connection alive for quite some time... I appreciate that might be too many "not-horrible" requirements for the real world today though.
Not on a Mac
Yes, this makes your connection more likely not survive client suspends. (ClientAliveInterval, which makes the server ping the client, will make it fail almost certainly, since the server will be active while the client is sleeping.)
Note this is only an issue if not using IPv6.
CGNAT is for access to legacy IPv4 only.
Well, for different reasons, but you have similar issues with IPv6 as well. If your client uses temporary addresses (most likely since they're enabled by default on most OS), OpenSSH will pick one of them over the stable address and when they're rotated the connection breaks.
For some reason, OpenSSH devs refuse to fix this issue, so I have to patch it myself:
The temporary address doesn't stay active while there's a connection on it? I think that would be the actual "fix".
I think it does, but that's not the issue: if the interface goes down all the temporary address are gone for good, not just "expired".
If you're on a stable address, and the interface goes down, will it let your connection/socket continue to exist?
Because if the connection/socket gets lost either way, I don't really care if the IP changes too.
I'm not sure what happens to the socket, maybe it's closed and reopened, but with this patch I have SSH sessions lasting for days with no issues. Without it, even roaming between two access points can break the session.
Interesting! Is there anywhere a discussion around their refusal to include your fix?
See this, for example: https://groups.google.com/g/opensshunixdev/c/FVv_bK16ADM/m/R...
It boilds down to using a Linux-specific API, though it's really BSD that is lacking support for a standard (RFC 5014).
It would also seem to break address privacy (usually not much of a concern if you authenticate yourself via SSH anyway, but still, it leaks your Ethernet or Wi-Fi interface's MAC address in many older setups).
This is a good argument for not making it the default, but it would be nice to have it as a command line switch.
Well, yss, but SSH is hardly ever anonymous and this could simply be a cli option.
Not anonymous, but it's pretty unexpected for different servers with potentially different identities for each to learn your MAC address (if you're using the default EUI-64 method for SLAAC).
This is a very common misconception. The issue is not IPv4 or CGNAT, it's stateful middleboxes... of which IPv6 has plenty.
The largest IPv6 deployments in the world are mobile carriers, which are full of stateful firewalls, DPI, and mid-path translation. The difference is that when connections drop it gets blamed on the wireless rather than the network infrastructure.
Also, fun fact: net.ipv4.tcp_keepalive_* applies to IPv6 too. The "ipv4" is just a naming artifact.
Mobile carriers usually have stateful firewalls for IPv6 as well (otherwise you can get a lot of random noise on the air interface, draining both your battery and data plan), so it's an issue just the same.
The constrained resource there is only firewall-side memory, though, as opposed to that plus (IP, port) tuples for CG-NAT.
> otherwise you can get a lot of random noise on the air interface, draining both your battery and data plan
I highly doubt you get "random" data over ipv6. There are more ipv6 addresses than there are atoms on the planet.
Yes, but they're not randomly distributed across the entire number space.
For example, receiving traffic from a given address is a pretty good indicator that there's somebody there possibly worth port scanning.
And where there has once been somebody, there or in the same neighborhood (subnet) might be somebody else, now or in the future.
Then it isn't random noise. It is determined by your own actions.
Or my predecessor/address space neighbor, or that of somebody using my wireless hotspot once, or that of me clicking a random link once and connecting to 671 affiliated advertisers's analytics servers...
I think a default policy of "no inbound connections" does makes sense for most mobile users. It should obviously be configurable.
putty is sending packets for network up since like forever
Have been using that weekly since probably 20 years. Will change your life :)
My other favourite is I very often SSH with -v to figure out why the connection is hanging, you rapidly figure out if DNS is failing, the TCP connection doesn't open, it does open but no traffic flows at all or it opens and SSH negotiation starts but never finishes. You can learn a lot just from this about what is wrong.
And of course, you can use the ~v / ~V commands (as listed in the ~? menu) to increase/decrease verbosity after the connection is established.
That lets you `ssh -vvvv` to a host then once you've figured out the issue use ~V to decrease verbosity so that debug messages don't clutter your shell.
Also helps with auth failures, I've used it several times with co-workers who can't figure out why their ssh key isn't working. It lists the keys out and some extra information.
You can even chain them if you have deep ssh connections (i.e. ssh from one instance to another). I think it would be ~~. to terminate the 2nd hop.
Edit: it's already explained in the OP
Be sure to hit enter before you start typing `~.`. It only works on a new line
Probably getting closer to 30 than 25 for me. And I used rsh before that where guess what was used to escape the input.
https://en.wikipedia.org/wiki/Berkeley_r-commands
> It's much nicer than my current approach of having to close that terminal window.
You can also just kill the ssh process (say from another terminal). That way you get to keep your terminal window. And this works with everything "blocking" your terminal, not just ssh.
You don't need to actually open the menu either. Just hit enter, tilde, ., enter.
If you regularly have to deal with hung connections or slow/unreliable links, I suggest trying out mosh.
https://mosh.org/
I last used this menu about 20 years ago when a dialup modem was the only way to roll, and have pretty much forgotten about it since the days of always-on direct to the desktop TCP/IP ..
Just ssh to funky.nondeterministic.computer to test it out!
I've been using ~. on hung ssh connections for a while.
I use that every day but it's the only one I know by heart lol
I've used ~. for a long time but did not know about others. I know, should have read man page.
Anyway, if you try it from shell prompt it is likely will not work as pressing ENTER shows the next prompt. Try `cat` followed by ENTER and then ~?
It'll still work. OpenSSH doesn't care about output (for ~ stuff), only input, so if you type <enter>~. it will close the connection.
Does not for me, not even with busybox sh and no funky escape codes in PS1 at all. It does with cat or yes running, so just something being output is not the problem… Hm.
It does not. open ssh linux to mac, typing ~ just types it on fish shell prompt. It works after`cat` followed by ENTER
Just type <enter> without cat, your shell will show you another prompt, and the ssh escape command will also work.
No they are correct, fish seems to intercept this or something like that. Only works with cat.
In newer versions, it's disabled by default and you have to do something like this to enable in ~/.ssh/config:
`EnableEscapeCommandline` only controls the <Enter>~C commandline.
The reason that is disabled in current OpenSSH by default is OpenBSD `pledge` support:
https://security.stackexchange.com/questions/280793/what-att...
On my Linux,
closes the connection as expected, and no ~ is shown in the terminal.Same with me, I'll still instinctively go for ~. when a connection has hung / dropped (usually because of a NAT via a rebooted firewall), but never even considered how ~ doesn't normally cause an issue. Never knew it had to be immediately following a newline. Also never knew about the other options, ~^Z in particular looks useful.
I wonder if anyone still remembers the ctrl-[ sequence in telnet. I think I only ever used the quit command in that though.
No need – your local ssh client, which interprets the escape sequence, doesn't have any understanding of whatever mode the remote session might currently be in.
The point of the return is to prime it to accept the start of a new escape sequence. Presumably the idea is that `~.` is not completely unlikely to occur as part of text entered remotely, but less so at the beginning of a new line.
Those aren’t “secret”, they’re obviously borrowed from rsh — oh that’s right, I’m old.
And these days you need to pass "-O" to scp to have it behave like rcp.
I'm pretty sure the ~ command style came from cu(1) which had it in at least 4.1BSD. I don't think rsh (which came in 4.2BSD) ever had such commands.
Whoops -- I meant rlogin. That had ~. and ~^Z. But you're right, rlogin got them from cu! (I'm not that old. :) )
Laughs in uucp!bangpath.
Yep the menu is handy for ssh tunneling. Maybe not a lot of people doing that these days though with stuff like dev tunnels and Tailscale.
I typically just create a "new" connection in a separate tab when I want to add tunneling.
I put new in quotes because I use another little-known feature, "ControlMaster". Multiplexes multiple connections into one, it makes making " new" sessions instant (can also be configured to persist a bit after disconnecting). Also useful for tab-completing remote paths. It does not prompt for authentication again, though. And it's a bit annoying when the connection hands (can be solved with ssh -o close, IIRC).
> I use another little-known feature, "ControlMaster". Multiplexes multiple connections into one, it makes making " new" sessions instant
Is this what secureCRT used as well? I remember this being all the rage back when I used windows, and it allowed this spawn new session by reusing the main one.
I'm using that as well but had issues with tunneling where it creates the tunnel in the background and terminates and so you might not know the random port it assigned or I couldn't figure out how to un-tunnel it and tunnel again to the same port. Just bypassed the control master then.
TIL; thanks, that's interesting (and somehow escaped my 20+ years of using ssh)! As usual the gold is in the comments :-)
I use it all the time with https://tuns.sh that let's you expose localhost to the public.
You think that one is cool, go check out "~?" in IPMI "sol activate". From there you can deliver a serial break to the kernel, which then gets you to a third rarely seen menu from the kernel's console, which allows you to do kernel debugging of various sorts.
I use it when I need to crash a kernel on purpose to test kdump over the network.
You can also send commands to the simulated console of a VM under libvirt with "virsh connect". But I don't think you can send a break to the kernel with that.
As secret as the ssh manual.
"secret" not in the sense that it's hidden, but that most people won't know about it. Because approximately nobody actually reads man pages in their entirety, they just get in to find out how a specific flag works and then get out.
Using "secret" in that sense instantly reminds me of hyped-up headlines for time-waster news stories. Most people don't know what 2^8 is, either.
the difference is that knowing 2^8 is generally not useful to people who don't know it
this here is something that's pretty useful to most ssh users, yet seldom spoken of
a better analogy would be comparing it to calling a very good, but not well-known restaurant a secret place - using the word to mean a hidden gem rather than an intentionally hidden secret
The ~ (tilde) escape character in UNIX cu (call UNIX) did the same thing in the 1970s, when the remote system was on the other side of a telephone modem or other serial port. I think it's safe to say that ssh is a descendant of cu. (I don't mean that cu printed a help menu, I mean it had the tilde escape.)
The ControlMaster mention deserves a full config block since the defaults make it awkward:
`mkdir -p ~/.ssh/sockets` first. With this setup, every subsequent ssh/scp/rsync to the same host reuses the existing master connection — no re-auth, near-instant open. ControlPersist keeps the socket alive for 10 minutes after the last session closes, so short gaps don't force a new handshake.This pairs well with the ~C escape discussed here: adding a port forward mid-session via ~C -L 8080:localhost:8080 doesn't require a new connection or re-authentication when a ControlMaster is already running. Useful for those "I need to tunnel something I didn't anticipate" moments.
Good tip on ControlMaster - also worth setting ControlPersist to something like 1h so the master connection stays alive between quick repeated logins.
Llm generated comment? Am I going crazy?
You are not crazy. This account's comments are all LLM generated. ALMOST believable
it starts with a pretty common char, but almost never gets in the way to the point I forget it exists. Meanwhile docker -t uses ^P which I use all the time for history instead of arrow keys. It's possible to configure it, but it's not worth the hassle on servers. Really, really annoying.
I've had to use [Enter] ~. in SSH sessions before. I've had SSH/network get stuck and using that was the only way to kill the session (and recover).
I guess you could call it a "secret" or at least "not super-well known (to people who aren't Linux 'experts')."
My gosh... I've spent decades closing a connection by killing the terminal.
~.
This is not specific to ssh. Telnet and rlogin have similar things with ~ as the escape character. Back in the day it was common to send BREAK and other escape sequences when you were hardwired.
Wow, never knew this... That said I'm not sure if I'll remember using it as my muscle memory is already trained to kill hung processes via ctrl-b s, ctrl-k, ctrl-b x (in tmux).
Yeah I discovered ~? a while back and had similar concerns, so I added:
into my .bashrcIt's like Ctrl + ] on telnet.
The good old times!
Unlike CTRL ], at least ~. doesn't require that I press two modifiers at the same time ... CTRL ALTGR $. Because people who define those kinds of shortcuts never consider how they might work on non-QWERTY layouts.
At least on FI/SE layout, ~ is a dead key. So it needs a space after it.
Still not as bad as brackets and infinitely better than {} on a macbook.
Try Ctrl-5.
Don't tell people this. In a minute you'll realize that this menu can only mean that ssh runs multiple channels, and so you can start up other things over an established ssh connection. Then the firewall guys will realize that they really can't allow ssh ... let's just not go there. Working at a bank is annoying enough already.
When your firewall guys "block" ssh (or anything else), all you need to do is run your ssh server on port 443 and then `ssh -p 443 user@host`.
Running on 443 means it will blend in with https traffic, making it much more difficult for them to detect and block (I've never actually seen anywhere that can detect/block it - you'd need to be doing deep packet inspection on encrypted traffic) :)
Nice. But is it possible to run both ssh and https (ideally with auto-certbotting) on the same port?
Not on the same IP at the same time, but you could always add a second IP for ssh. Just configure sshd to only listen on one IP and <webserver> to listen on the other.
This might be a good place to mention: https://linux.die.net/man/1/autossh
https://infosec.exchange/@rebane2001/116200045748516097
BRB requesting access to my remote server "animal style"
Secret Menu -> Escape Characters
I really hate it when people just rename terms. It made it harder to search properly for better answers.
https://xkcd.com/1053/
I have set up Ctrl+k to kill (SIGKILL) the front process in my terminal. I just use that for hanging connections and other hanging processes.
What I thought I wanted: a way for Konsole to send SIGHUP
What I now have: ~B
What I really need: a way to stop long-running SSH connections from freezing
This is a secret? I've had this baked into my muscle memory for decades when an SSH session hangs.
Sometimes things feel so simple that i dont even read the manual. cool
FYI, you can kill forwarding tunnels with -KD portnum in that commandline too.
That doesn't do much good if you set `EscapeChar` to `none` in `.ssh/config`.
I find it convenient not to have to worry about accidentally entering escape characters. YMMV.
Note that it only works after pressing enter, so the odds are slim. In practice, I don't think I ever hit it by accident.
I have noticed it while running ~/bin/some_command. The ~ doesn't echo until I also type the /. It doesn't cause any misbehavior because there is no binding for ~/ but can be slightly surprising.
I find it odd that you would have commands in ~/bin but not have it be the highest priority in your PATH. I use ~/.local/bin, but would never type it because i wouldn't have bins that overlap shell commands and no other path would have priority.
Usually, it is. IIRC, this was when I was just setting up my environment on a new host, after I had populated ~/bin but before I restarted my shell to pick up PATH modifications.
SSH does it pretty well though. Never once have I done it by mistake.
I'd guess this is because it only works in ssh PTY sessions. So it would have no effect on tunneling or when piping arbitrary data through ssh to a non-interactive remote command (unless you use the -t switch to force PTY allocation even when stdin is not a TTY).
No I don't think so. I mainly and pretty much constantly use SSH for logging in. I'm not one of those 'cattle not pets' guys lol.
And when I port forward I usually don't even tunnel it over SSH because all my stuff is on tailscale so it's also encrypted.
consider me one of the lucky 10000 . Used ssh for years, didn't know this!
https://xkcd.com/1053/
`ssh -D 1234` to open up SOCKS in the server. This is also flew over my head. I think the gzip one is the most surprising tbh.
I use that one a lot! It is extremely useful for accessing web-interfaces on switches and firewalls and other devices on a remote network.
Find the HIDDEN SECRETS that THEY DON'T WANT YOU TO KNOW!
Many Linux man pages have the thoroughness of a fortune cookie, so I can understand the skepticism.
Jackpot if they're just a pointer to an 'info' page.
Most people just discount man pages as unreadable and don’t even try to understand them.
Case in point: the jq man page is incredible and everyone I know instead runs off to google or stackoverflow or Claude to answer simple questions
The real jackpot is if they're the same as the --help command
I used to think this and used things like `help2man`. I now disagree, but throwing it out there.
I could use some "help2man." I don't know how to "man" /s
But its also true that many, many man pages have extremely valuable information that no enterprising hacker should overlook, too ..
I was never able to properly parse large man pages, I'm so happy that llms can now prepare half a usable command without spending an hour reading a time without a single usage example.
What I usually do when I have to read large man pages like bash(1) is I read them as PDFs:
man -Tpdf bash | zathura -
Replace zathura with any PDF viewer reading from stdin or just save the PDF. Hope that can be useful to someone!
my manpager is `vim -`, can't beat that
You probably can — by using neovim:
https://wiki.archlinux.org/title/Neovim#Use_as_a_pager
https://neovim.io/doc/user/filetype/#_man
I've also been running (neo)vim as a manpager. You get the same features as with vim (like easily copying text or opening referenced files/other manpages without using the mouse), but neovim also parses the page and creates a table of contents, which can be used for navigation within the page. It doesn't always work perfectly, but is usually better than nothing.
`tldr` is also great. It's essentially a collection of example invocations of *nix commands.
TIL that what I have wanted in manpages for years exists. Thank you!
tldr is so good, i wish it was a part of the os the same way manpages are just to help out newcomers
Do we still have those? I think it was common in late '90s, due to GNU trying to get `info` gain moment but nowadays?
Mostly just the bash builtins have (had) this problem.
I blew a few minds a week ago when I told my younger coworkers about man pages. I hit ‘em with the `man man` and one dude was like “whoa!”
Further blow their minds by showing them `apropos` https://manpages.debian.org/testing/man-db/apropos.1.en.html
I remember my first days with Linux/Unix and being taught about man and apropos and info. But somehow I could never find anything useful via apropos!
With many years of insight, I think I probably never updated the database.
man -k, apropos, but less to type
I'm 100% certain this is user error, but I have not once gotten apropos to give me any output other than "nothing appropriate."
Reminds me of the old canard of 'man woman'
man, cat, lynx... We're all just creatures in a vast universe
$ man -k <whatever>
Who doesn’t want you to know? Well, obviously, the man. That’s why you type man ssh, you’re forcing the man to tell you what he knows.
man ssh_config is even more interesting and hidden
ProxyCommand is fun
You can also run commands when a user authenticates, grab their keys from github.com/username.keys, validate they're a user in a specific github group, then let them connect by outputing the keys, otherwise nothing to deny them access.
It's really great for ops teams where you want to give ssh access and manage it from github teams without needing a complex system.
Honest question, why is ProxyCommand `fun`? What do I get out of ProxyCommand that i do not get out of setting the correct order for ProxyJump and doing an ssh finalhost -- domy --bidding?
ProxyJump is a newer functionality. There used to be only ProxyCommand. ProxyJump is a shortcut for the usual way to use ProxyCommand to connect through a bastion host but ProxyCommand is more flexible. For example with ProxyCommand you can run any command to connect to the remote host. ProxyJump only connects over ssh. I think I replaced all my ProxyCommand with ProxyJump because I don't need much else than the normal use case.
You can get a lot more out of ProxyCommand. For example, you can run SSH over non-IP protocols, such as serial, Bluetooth RFCOMM for embedded boards, or vsock for virtual machines without networking set up at all. The latter is built into and setup up automatically by systemd:
https://www.freedesktop.org/software/systemd/man/257/systemd...
Much appreciated!
ProxyCommand allows you to use any command to setup a connection. Not necessarily an ssh command, like ProxyJump. It can be any command, as long as it receives on stdin and produces on stdout, it can act like a TCP connection.
ProxyJump is a special case of `ProxyCommand ssh -p <port> <user>@<host>`. Can't replace the `ssh` in there when using ProxyJump.
I came across ProxyCommand earlier this week, funnily enough. I have Cloudflare Zero Trust set up with an SSH service[0], and have the server firewall drop all incoming traffic. That helps reduce my attack surface, since I don't have any incoming ports open.
[0]: https://developers.cloudflare.com/cloudflare-one/networks/co...
I use ProxyCommand in edge-case devices where key auth is not an option and the password is not controlled by me. ProxyCommand points to a script the retrieves the password from the vault, puts it on the clipboard for pasting, reminds me via stderr it's done so, and then proxies the connection.
Interesting. I might have such a use case. Do you have anything about best practices on how to automate grabbing passwords from vaults? Cuz it seems to me that the vault needs to be kept open or keep the vault password somewhere on disk.
I use ProxyCommand to run spipe tunnels for SSH.
`pgrep`/`pkill` HATE this trick! Learn how this renegade developer (relatively) easily exits their hanging SSH sessions without restarting their laptop.
Not as much as they hate Enter ~ .
See "escape characters" under man ssh.
Shh, you're giving away the trick for free! Please think of my clickthrough rates and ad revenue.
I still struggle with the fact that I was (or delusional) an effective C programmer "back in the day" (before google etc) and all we had was "man" to look up std/x11 system calls.
Now I am dismayed with juniors who can't even be bothered to use google (or llms) to look up stuff on their first hiccup.
#include <old-man-shouting-at-clouds>
That’s not a builtin, so wouldn’t it be:
And it's kinda old, so maybe
Well played good sir, well played
HN as a tabloid ... After all enquiring minds want to know!
It’s been a while since I’ve heard RTFM
RTFM gave way to LMGTFY, and I next is something like ATLLM (ask the LLM)
Because it is associated with "toxicity". Yeah, I know.
They're sticking it to the man
Working for the man eh?
sssshhhh...don't just give out secrets like that .
hahaha made me laugh, thanks :-D
How's that supposed to help? The ssh man page is about as close as you can get to a Platonic example of "uninformative pretend-documentation".
If the ssh man page were any lazier it would just be a list of arguments with no context at all, so I get the frustation. The tricky bit is that much of the "secret menu" knowledge about ssh only lives in blog posts and random issue comments, which makes the doc gap feel intentional at times. If you want real examples or usage patterns you pretty much have to scrape Stack Overflow or crawl through dotfiles in public repos hoping someone else already fought the same fight.
[dead]
Hidden or undocumented features like this always have a strange appeal. Part of it is nostalgia for older software where small Easter eggs or experimental features would sometimes ship in production builds.
It's not hidden or undocumented; it's in the man page.
Here's 15-year old HN link about it: http://grack.com/blog/2011/02/23/ssh-escape-sequences-or-don...
Here’s a link to the man page for people who want to read man pages in a browser. https://linux.die.net/man/1/ssh
https://die.net and https://ss64.com are sites I’ve been recommending for years.