SSH is yet another example of an ancient technology that is still in wide use today. It may very well be that learning a couple of SSH tricks is more profitable in the long run than mastering a dozen Cloud Native tools destined to become deprecated next quarter.

One of my favorite parts of this technology is SSH Tunnels. With nothing but standard tools and often using just a single command, you can achieve the following:

• Access internal VPC endpoints through a public-facing EC2 instance.
• Open a port from the localhost of a development VM in the host's browser.
• Expose any local server from a home/private network to the outside world.

And more 😍

But despite the fact that I use SSH Tunnels daily, it always takes me a while to figure out the right command. Should it be a Local or a Remote tunnel? What are the flags? Is it a local_port:remote_port or the other way around? So, I decided to finally wrap my head around it, and it resulted in a series of labs and a visual cheat sheet 🙈

SSH Tunnels

What is SSH tunneling?

SSH tunneling is a method to transport additional data streams within an existing SSH session. SSH tunneling helps achieve security use cases such as remote web service access without exposing port on the internet, accessing server behind NAT, exposing local port to the internet.

How does the SSH tunnel work?

When you connect to a server using SSH, you get a server's shell. This is the default behavior of an SSH connection. Under the hood, your SSH client creates an encrypted session between your SSH client and the SSH server. But the data transported within the SSH session can be of any type. For example, during shell access, the data transmitted are binary streams detailing dimensions of pseudo-terminal and ASCII characters to run commands on the remote shell. However, during SSH port forwarding, the data transmitted can be a binary stream of protocol tunneled over SSH (e.g. SQL over SSH).

An SSH Session

So SSH tunneling is just a way to transport arbitrary data with a dedicated data stream (tunnel) inside an existing SSH session. This can be achieved with either local port forwarding, remote port forwarding, dynamic port forwarding, or by creating a TUN/TAP tunnel. Let's take a look at how port forwarding works and their use cases below.

SSH Local Port Forwarding

Starting from the one that I use the most. Oftentimes, there might be a service listening on localhost or a private interface of a machine that I can only SSH to via its public IP. And I desperately need to access this port from the outside. A few typical examples:

• Accessing a database (MySQL, Postgres, Redis, etc) using a fancy UI tool from your laptop.
• Using your browser to access a web application exposed only to a private network.
• Accessing a container's port from your laptop without publishing it on the server's public interface.

All of the above use cases can be solved with a single ssh command:

ssh -L [local_addr:]local_port:remote_addr:remote_port [user@]sshd_addr

The -L flag indicates we're starting a local port forwarding. What it actually means is:

• On your machine, the SSH client will start listening on local_port (likely, on localhost, but it depends - check the GatewayPorts setting ).
• Any traffic to this port will be forwarded to the remote_private_addr:remote_port on the machine you SSH-ed to.

Here is how it looks on a diagram:

SSH Local Port Forwarding with a Bastion Host

It might not be obvious at first, but the ssh -L command allows forwarding a local port to a remote port on any machine, not only on the SSH server itself. Notice how the remote_addr and sshd_addr may or may not have the same value:

ssh -L [local_addr:]local_port:remote_addr:remote_port [user@]sshd_addr

Not sure how legitimate the use of the term bastion host is here, but that's how I visualize this scenario for myself:

I often use the above trick to call endpoints that are accessible from the bastion host but not from my laptop (e.g., using an EC2 instance with private and public interfaces to connect to an OpenSearch cluster deployed fully within a VPC).

SSH Remote Port Forwarding

Another popular (but rather inverse) scenario is when you want to momentarily expose a local service to the outside world. Of course, for that, you'll need a public-facing ingress gateway server. But fear not! Any public-facing server with an SSH daemon on it can be used as such a gateway:

ssh -R [remote_addr:]remote_port:local_addr:local_port [user@]gateway_addr

The above command looks no more complicated than its ssh -L counterpart. But there is a pitfall...

By default, the above SSH tunnel will allow using only the gateway's localhost as the remote address. In other words, your local port will become accessible only from the inside of the gateway server itself, and highly likely it's not something you actually need. For instance, I typically want to use the gateway's public address as the remote address to expose my local services to the public Internet. For that, the SSH server needs to be configured with the GatewayPorts yes setting.

Here is what remote port forwarding can be used for:

• Exposing a dev service from your laptop to the public Internet for a demo.
• Hmm... I can think of a few esoteric examples, but I doubt it's worth sharing them here. Curious to hear what other people may use remote port forwarding for!

Here is how the remote port forwarding looks on a diagram:

SSH Remote Port Forwarding from a Home/Private Network

Much like local port forwarding, remote port forwarding has its own bastion host mode. But this time, the machine with the SSH client (e.g., your dev laptop) plays the role of the bastion. In particular, it allows exposing ports from a home (or a private) network your laptop has the access to to the outside world through the ingress gateway:

ssh -R [remote_addr:]remote_port:local_addr:local_port [user@]gateway_addr

Looks almost identical to the simple remote SSH tunnel, but the local_addr:local_port pair becomes the address of a device in the home network. Here is how it can be depicted on a diagram:

Since I typically use my laptop as a thin client and the actual development happens on a home server, I rely on such a remote port forwarding when I need to expose a dev service from a home server to the public Internet, and the only machine with the gateway access is my thin laptop.

SSH Dynamic port forwarding

Both local and remote port forwarding require defining a local and remote port. What if the ports are unknown beforehand or if you want to relay traffic to an arbitrary destination? Also known as dynamic tunneling, or SSH SOCKS5 proxy, dynamic port forwarding allows you to specify a connect port that will forward every incoming traffic to the remote server dynamically.

Dynamic port forwarding turns your SSH client into a SOCKS5 proxy server. SOCKS is an old but widely used protocol for programs to request outbound connections through a proxy server.

Bypassing content filters - By tunneling network traffic inside SSH, it is possible to access services such as HTTP web pages that may be blocked by your ISP or organization. SOCKS5 proxy can proxy any type of traffic and any type of protocol. This means you can tunnel HTTP inside SSH using SOCKS5.

SSH dynamic port forwarding

Consider the case above. The client cannot access certain web pages due to content filtering implemented in front by the firewall. But the client can connect to the remote server (via SSH), which has unrestricted access to the internet. In this case, the client can initiate ssh connection by enabling dynamic port forwarding, thus creating a SOCKS5 proxy; the client can point web browser to proxy listening on local port 6000 and access restricted web content.

Command used for dynamic SSH tunneling scenario above:

ssh -D 127.0.0.1:6000 user@<remote_server>

Summarizing

After doing all these labs and drawings, I noticed that:

• The word "local" can mean either the SSH client machine or an upstream host accessible from this machine.
• The word "remote" can mean either the SSH server machine (sshd) of an upstream host accessible from it.
• Local port forwarding (ssh -L) implies it's the ssh client that starts listening on a new port.
• Remote port forwarding (ssh -R) implies it's the sshd server that starts listening on an extra port.
• The mnemonics are "ssh -L local:remote" and "ssh -R remote:local" and it's always the left-hand side that opens a new port.

Resources

• original article from iximiuz
• SSH Tunneling Explained by networking gurus from Teleport.
• SSH Tunneling: Examples, Command, Server Config by SSH Academy.