SSH Tunnels and Systemd

Posted by Nate Bargmann on Sun, Mar 19, 2023

OpenSSH, hereinafter simply called SSH, is a protocol and set of programs most commonly known for secure access to remote computers that often is used to perform remote administration at a shell prompt, i.e. Command Line Interface (CLI). In addition, SSH can work as a transport for other programs such as rsync which are not secure in and of themselves. In this way one can be assured of actually connecting to the intended remote host based on key exchange and that the data transfer will be secure via SSH encryption. Another feature are the creation of tunnels to access services on a remote host that you may not want to expose to the Internet such as a Web server. This feature will be the subject of this post.

By now systemd is the de facto standard for service management and other functions of a modern Linux system. This post will show examples of systemd unit files to start and stop an SSH tunnel. The needed systemd command will be embedded in a set of Bash shell aliases for ease of use at the shell prompt.

On its own SSH is an exhaustive subject and I will not be covering all steps in this post but will refer to other resources. One of the primary resources will be the venerable Arch Linux Wiki. Among the prerequisites is knowledge of using SSH with a normal password and also using public key authentication. Advanced readers will be able to set up a client and server for public key authentication and disabling password authentication.

A final component is the use of Autossh to establish and maintain the persistence of an SSH tunnel. As a resource I used SSH tunnelling for fun and profit: Autossh which is part 3 of a series found at that site. Do use the menu on that page and read all four parts.

Getting started

I will assume you have two computers on your LAN and that you’ve set up public key authentication between them and have disabled password authentication. I will use the terms “client” and “server” in the sense that the client is the computer the SSH connection is being established from and the server is the computer receiving the connection. Sometimes this distinction becomes very confused!

The resources above should have shown setting up the SSH configuration file for common settings on the client for the server. Here is a simple $HOME/.ssh/config entry (assuming you have DNS on your LAN that resolves local host names, if not, an IP address may be used):

Host server.lan
    User eastwood
    IdentityFile /home/clint/.ssh/id_server

As usual, the ssh_config manual page (man ssh_config at a shell prompt) is a quick reference for these settings.

With this clint can simply type ssh server.lan and SSH will read this file, find the server.lan entry and contact the server to establish the SSH connection, prompt clint for the password to the key named id_server, and upon entry of the correct password, log clint into server.lan as the user eastwood.

While that is easy enough, clint will be setting up connections to a number of servers and opts to create a Bash shell alias as a shorthand for ssh server.lan and adds it to his $HOME/.bashrc file so it is always available for interactive shell sessions:

alias ssrvr='ssh server.lan'

While this is a small gain of efficiency, a short mnemonic is often simpler than using the command history search capability of Bash.

Some free desktop environments (GNOME is one such) start a key agent that will offer to cache the keys and unlock them when logging into the desktop. Beware that this convenience can lead to forgotten key passwords! I’ve found it useful to store such passwords securely. A key agent increases efficiency by not having to type the password each time clint opens an SSH connection to server.lan.

SSH tunneling

At some point there is an intention to deploy a system accessible over the Internet but for various reasons the Web server listening on port 80 will be blocked by the system’s firewall from being accessed from the Internet. While there are certainly alternatives, this post focuses on using an SSH tunnel to securely access the remote Web server. Be sure to read through and understand SSH tunnelling for fun and profit: local vs remote for background.

Building on the SSH configuration above, the following stanza is added:

Host server-httpd
    HostName server.example.org
    Port 22
    User eastwood
    IdentityFile /home/clint/.ssh/id_server
    LocalForward 12380 localhost:80
    ExitOnForwardFailure yes
    ServerAliveInterval 30
    ServerAliveCountMax 3

The HostName option is only required if Host does not resolve to an IP address. In my file I use the Host option to separate configurations for different servers or services on a given host.

The Port option is only required if server is listening on a different port than the default 22.

The LocalForward option is a bit confusing. The first value given here, 12380, is the TCP port on the local machine through which the remote Web server will be accessed, i.e. by typing ‘localhost:12380’ into the browser’s address bar. The second value is the host and port that server is to connect the tunnel. In this case it is server itself that will receive the connection through the network loop back interface, a.k.a. localhost at TCP port 80, the standard port for the Web server.

Note: There are three classes of TCP ports, system, user and dynamic. System ports are those numbered from 0 to 1023, user ports are those from 1024 to 49151, and dynamic ports are those from 49152 to 65535. System ports require root (administrator) privileges for a process to open for listening while user ports can be opened by any user on the system. Many user ports are assigned for specific services, although such a particular service may not be running on your system.

A list of common service ports can be found in /etc/services and to see what ports are in use on your system use the following command:

sudo netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      3627221/cupsd       
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1744/exim4          
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      1406/inetd          
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      869/sshd: /usr/sbin 

You can see various services running on the system, the address they are bound to—localhost or the default or any address. So long as a user port number is chosen that is not in use, the tunnel should be configured which can be confirmed by running the netstat command again.

Assuming that public key authentication has already been configured for server, open an SSH connection to it using the tunnel configuration:

ssh server-hhtpd

Did it work? You probably saw that you logged into server and received a shell prompt. In your terminal emulator open another tab (or terminal if using XTerm or rxvt) and run the netstat -tunlp command and see if there is an entry like this:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:12380         0.0.0.0:*               LISTEN      3897982/ssh

Try connecting to localhost:12380 with your Web browser, you should see the default page. When you exit the shell on server the tunnel is also closed. Verify this by reloading the page in the browser.

For the most part, when creating a tunnel for a service logging into a shell prompt is not needed. SSH provides a couple of options for just this purpose. The -N and -T options instruct SSH to not execute a remote command and to not allocate a pseudo terminal, respectively:

ssh -N -T wxbox-httpd

Now the command appears to hang and do nothing but a quick check with netstat shows the tunnel is present:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:12380         0.0.0.0:*               LISTEN      4105760/ssh         

Everything looks the same as above but with a much different PID for this instance of SSH. The browser is able to load the page from server. Now the ^C (Control-C) key combination must be used to exit the tunnel. It can now be verified that the tunnel has been closed and removed.

Autossh and systemd

The idea of setting up an SSH tunnel is to have a persistent link to a remote service. Many things can cause the SSH link to fail. One solution is Autossh which is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic. (from its manual page).

Pair Autossh with systemd and we have a powerful way to start and stop a tunnel, though in this article I will configure systemd through its user capability rather than its system capability.

On Debian and Arch systems systemd is configured to load any unit files it finds in several places under the user’s home directory. Unit files created by the user should be placed under $HOME/.config/systemd/user with the extension of .service. For this example I will name it server-httpd.service (imaginative, huh?):

[Unit]
Description=AutoSSH tunnel service to server-httpd via server.lan on port 12380
Wants=network-online.target
After=network-online.target nss-lookup.target

[Service]
ExecStart=/usr/bin/autossh -M 0 -N -T server-httpd

[Install]
WantedBy=default.target

The ExecStart line probably looks the most familiar as it is much like the ssh commands above. The -M 0 option uses the ServerAliveInterval and ServerAliveCountMax option values set in the SSH configuration file above to have autossh restart the connection should SSH exit. The rest is rather standard systemd boilerplate.

If you’ve never set up a systemd user unit file before then the following command may fail:

systemctl --user status server-httpd.service

and you need to run this command:

systemctl --user daemon-reload

for systemd to recognize the new unit file. Otherwise, you should see output like:

 server-httpd.service - AutoSSH tunnel service to server-httpd via server.lan on port 12380
     Loaded: loaded (/home/user/.config/systemd/user/server-httpd.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

The Loaded line is what you want to see. The fact that the unit is disabled is not a problem as my understanding is that indicates whether or not the unit will be run automatically. For this exercise we don’t want it to start automatically. Instead, this command will start the service manually:

systemctl --user start server-httpd.service

The output of netstat should show the tunnel is active:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:12380         0.0.0.0:*               LISTEN      690392/ssh          

which can be confirmed with loading the Web page in the browser.

The tunnel can be stopped:

systemctl --user stop wxbox-httpd.service

and confirmed as well.

Finally, a pair of Bash aliases can be created as a mnemonic for easy recall:

alias srvr-httpd-start='systemctl --user start server-httpd.service'
alias srvr-httpd-stop='systemctl --user stop server-httpd.service'

Of course, these aren’t absolutely necessary as the shell’s command recall capability makes it easy to find such used commands. However, often the command history will be truncated in its default configuration and the previously used commands may be lost unless they’re used frequently. If nothing else, setting an alias is a form of permanent storage for these commands.

Despite the similar names, the shell’s tab completion capability allows typing a few letters, pressing the Tab key, and the shell will complete the command up to the point where the commands differ. Typing the differing letter—a or o in this case—and pressing Tab again will complete the command.

Summary

SSH tunnels are a very useful feature and with Autossh and systemd persistent tunnels can be easily started and stopped as needed. Peace of mind comes from being able to interact with a remote host without exposing it to the Internet, at least with no more exposure than an open SSH port.