Basics

The ssh program is used to connect to remote machines. There are many helpful articles online:

  1. https://linux.die.net/man/1/ssh from the manual: you should read the manual ;-)
  2. https://www.digitalocean.com/community/tutorials/how-to-use-ssh-to-connect-to-a-remote-server
  3. https://www.baeldung.com/cs/ssh-intro

Several of the HPC centres our scientists work on also have internal SSH documentation:

In practice, you want the following:

$ ssh <USER>@<REMOTE_HOST>

One of the most common flags is either -Y, -X, which enables X-Window forwarding. You can also configure ssh via a configuration file, which you can read about here.

Keys and the ssh-agent

Using ssh program to connect to remote machines is a common pratice in a HPC environment. Typically, HPC sites require user to generate ssh-key so that authentication is carried out based on this pre-shared public ssh-key. The private and public ssh-keys reside on users local machine (laptop or desktop) in their home directory and is not required to copied to the remote machine (HPC login node). A user having accounts at multiple HPC sites can have unique ssh-key paris for each one of them.

In the simpler usecase, connection to any of these remote HPC sites from local machine is straightforward as the ssh-keys are availble on the local machine for authentication but what if the requirement is to establish connection between two remote machines, say for data transfer between them. How does the work-flow looks like in this case? One naive approach is to copy the ssh-keys from local machine to one of the machines so that authencation credentails are available on that machine to connect to the other machine. The same can be done on the other remote machine if the connection from that machine is required to the other remote machine. what if there are 10 different remote HPC machines, the naive approach does work but it soon becomes cumbersome to mainting ssh-keys at different sites. Also what if one of the remote sites is compromized? As ssh-keys are available on that remote site, the unauthorized user can use these keys to connect to the other remote sites without our knowledge. This poses a huge potential security risk. A better approch to this problem is to use ssh-agent.

Depending on the operating system, ssh-agent service may or may not be turned on by default. Assuming the ssh-agent service is running, one can add ssh-keys to the agent. ssh-agent can forward the keys for authentication when establishing the connection without copying keys to remote machine. It can also authenticate from one remote machine to the other remote machine without having to copy the ssh-keys to any of the remote machines.

To ensure ssh-agent is running, check out the environment variable:

$ echo $SSH_AUTH_SOCK
/private/tmp/com.apple.launchd.rrClL8DMZH/Listeners

To add a specific key to the agent:

$ ssh-add -i ~/.ssh/id_ed25519

To add all the keys to the agent:

$ ssh-add -A

To list all the keys loaded:

$ ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1...nmyx AWI
ssh-ed25519 AAAAC3NzaC1lZDI1...nq3a DKRZ

ssh-ing to albedo with key-forwarding. If -A option is not provided the ssh-agent does not forward the keys.

$ ssh -A pasili001@albedo0.dmawi.de
Last login: Tue Nov  7 15:08:42 2023 from x.x.x.x
pasili001@albedo0:~$ 
pasili001@albedo0:~$ ls -al ~/.ssh
total 3
drwx--S--- 2 pasili001 hpc_user 4096 Nov  3 18:58 .
drwx--S--- 7 pasili001 hpc_user 4096 Nov  7 17:16 ..
-rw------- 1 pasili001 hpc_user  102 Nov  2 12:24 authorized_keys
-rw-r--r-- 1 pasili001 hpc_user 2348 Nov  7 15:12 known_hosts

Observe that there is no ssh-keys pairs in this folder (~/.ssh). As the ssh-keys were forwarded (ssh -A) while connecting to albedo, it should be possible to connect to levante (another remote HPC site) from albedo directly.

pasili001@albedo0:~$ ssh a270243@levante.dkrz.de
Last login: Mon Nov  6 11:37:56 2023 from x.x.x.x
[a270243@levante1 ~]$
[a270243@levante1 ~]$ logout
Connection to levante.dkrz.de closed.
pasili001@albedo0:~$

copying a file from levante to albedo

pasili001@albedo0:~$ cd tmp
pasili001@albedo0:~/tmp$
pasili001@albedo0:~/tmp$ scp a270243@levante.dkrz.de:~/tmp/sample.py .
sample.py                             100%  637    34.5KB/s   00:00
pasili001@albedo0:~/tmp$

using rsync for the transfer. Requires specifying flag (--rsh="ssh -A") so rsync is made aware of the forwarded ssh-keys.

pasili001@albedo0:~/tmp$ ls
sample.py
pasili001@albedo0:~/tmp$ rsync -av --rsh="ssh -A" a270243@levante.dkrz.de:~/tmp/pyworks .
receiving incremental file list
pyworks/
pyworks/analyse.py
pyworks/fesom_pool.py
pyworks/listings_albedo.csv
pyworks/listings_levante.csv

sent 123 bytes  received 35,655 bytes  23,852.00 bytes/sec
total size is 35,239  speedup is 0.98
pasili001@albedo0:~/tmp$ ls
pyworks  sample.py

Login with a Hardware Security Key

Hardware security keys are a form of Multi-Factor Authentication (MFA) to increase security on logins. Logins require of the hardware to be plugged in into your local machine and that you activate them by touching them. An example of such hardware are the YubiKeys that uses FIDO2 authentication standard, supported by OpenSSH (except for the MacOS version).

How to create an ssh-key that uses a YubiKey?

To use a YubiKey to login into Albedo you need to follow a very similar procedure as to setup a standard ssh-key:

  1. Connect the YubiKey to your local computer.

  2. Create an ssh-key using the type ed25519-sk (the sk is what tells ssh-keygen that you want to use a hardware security key), and -a 100 for the number of key derivation function rounds (for increased security):

    ssh-keygen -a 100 -t ed25519-sk -f ~/.ssh/id_ed25519_sk_<a_string_of_your_choice>

  3. If you have a recent version of OpenSSH, it should ask now for you to touch the button in the key. If it doesn't you probably need to update your OpenSSH client, or install OpenSSH (e.g. in MacOS using for instance homebrew).

  4. It will also ask for a passphrase. We strongly recommend to also use a passphrase for increased security. Passphrasses can be store in the ssh-agent (as with ssh-keys that do not use hardward security keys), so that you need to enter the passphrase less often.

  5. Transfer the public to Albedo, for example by using:   

    ssh-copy-id -i ~/.ssh/id_ed25519_sk_<a_string_of_your_choice>.pub <your_username>@albedo1.dmawi.de 

Additionally you can add a configuration for this newly created ssh key in ~/.ssh/config as explained here: https://spaces.awi.de/spaces/HELP/pages/394243582/Beginners+guide

How to login in Albedo using the already-configured YubiKey?

Login as you would login with any other ssh-key, using the id_ed25519_sk key. Make sure the key is plugged in into your local computer before you try to login. Once you execute the ssh command it should ask you to touch the YubiKey button. If you introduced a passphrase it will also ask you for a passphrase. Passphrases could be remember via the ssh-agent, but Yubikeys will always be asked every time you try to login.