module avail

Compiler & MPI

We currently provide the following compilers:

Compiler options for optimization

Some remarks from Natalja with respect to the intel compiler and the FESOM2 benchmark (used by NEC in there albedo offer):

1. do not  use -xHost, because Intel does not "recognize" AMD (officially for security reasons . Therefore use: -xcore-avx2
2. These compiler options were used by NEC during the FESOM2 benchmark:

   OPT = -O3 -qopt-report5 -no-prec-div -ip -fp-model=fast=2 -implicitnone -march=core-avx2 -fPIC –qopenmp -qopt-malloc-options=2 -qopt-prefetch=5 -unroll-aggressive

   These are (at least partially) quite important for good performance. However, we do not have the experience which are more or less critical. Be careful, some options might kill reproducibility (e.g., -fp-model=fast=2).
3. Natalja is now responsible for this: https://docs.dkrz.de/doc/levante/running-jobs/runtime-settings.html#open-mpi-4-0-0-and-lat let's stll benefit from here knowledge
4. Independent on the MPI used, please try runtime setting
   UCX_TLS=knem,dc_x,self
for your jobs. According to NEC for smaller Jobs it might be beneficial to replace "dc_x" with "rc_x".
5. When using OpenMPI, parallel IO (using OMPIO) might be slow. Try using
   

   export OMPI_MCA_io="romio321"

Spack

On albedo we mainly use spack to install software and provide module files.

On albedo it can also be used by users to

• install their own software into their $HOME
• load specific packages (similarly to what environment modules do)

Simply load the spack module:

module load spack

This version is configured, such that it makes use of the global software tree, installed by the admins, and your installations go into $HOME/.spack .

Please consult the official documentation on how to use spack: https://spack.readthedocs.io

Python, R, conda, Jupyter

Python

We provide Python modules with only basic Python installed for currently supported Python versions 3.7 through 3.10. Further packages you might need on top of this can be installed via standard methods, e.g. pip.

Additionally, we provide a toolbox of common data analysis tools in both Python and R. This module is available under analysis_toolbox and is updated every 3 months. This module is actually a conda environment which simply sets the correct shell variables (e.g. $PATH, $PYTHONPATH, and similar) for you. A list of currently installed Python and R tools in this module may be found under /albedo/soft/conda-workspace/env-ymls/analysis-toolbox-03.2023.yml.

If you would like additional libraries in this globally available analysis toolbox, just ask! They will be added whenever the next one is released (January, March, June, and September)

R

Similar to Python, the some basic R modules are included in the analysis_toolbox. Currently, there is no R-Studio available but we are planning to install it in the September-2023 update of the analysis_toolbox. Be aware that we still recommend against using graphical interfaces since that's not what an HPC is designed for. Our recommended workflow for using R is either with the interactive session and the R, or via Rscript and sbatch script:

Interactive session with R command line interface

From Albedo's login node:

salloc --account=<your_account> --time=<HH:MM:SS> --qos=<QOS> --nodes=<#Nodes> <other_options...>

To understand which options you need to specify please refer to the Jobs section on Albedo-Slurm and the SLURM user guide.

module load analysis-toolbox
R

The R command line will open and you can start using R from there.

Rscript and sbatch script

From Albedo's login node:

$ module load analysis-toolbox

The command Rscript allows you to run an R script you wrote from the shell (outside of R IDE or R-Studio). This means that you can also write a sbatch script (see Albedo-Slurm > Jobs) that runs your R scripts via Rscript and submit it to the slurm queue using sbatch command. You can even use the slurm array feature to launch the same R script multiple times in parallel if what your script does can be broken into multiple computing chunks. Depending on your R script it might need more or less changes, but it's probably worth to spend the time on chaning it to be able to benefit from this first order parallelization. For a simple example on slurm array + Rscript the following tutorial covers most of it: https://rcpedia.stanford.edu/topicGuides/jobArrayRExample.html

Conda

Conda is a package manager for Python, R, and Julia software. You can use it on our HPC system by:

$ module load conda

Thereafter, you should be able to use conda to manage your Python/R/Julia environments. Typically you will want to install software via:

$ conda install -c conda-forge <PACKAGE_NAME>

Note that this allows you to install both Python as well as R packages. Full documentation for conda is provided here: https://docs.conda.io/en/latest/ 

A useful cheat sheet for using conda can be found here: https://docs.conda.io/projects/conda/en/4.6.0/_downloads/52a95608c49671267e40c689e0bc00ca/conda-cheatsheet.pdf

Jupyter

Jupyter is an interactive computing environment which lets you execute notebooks which can mix code, text, graphics, and LaTeX all in a single document. There are different ways to use Jupyter from Albedo listed bellow.

JupyterHub

See Jupyterhub on Albedo.

JupyterLab from a login node

Please, read until the end of this section, the last step is really important for things to work. Load the analysis-toolbox:

[mandresm@albedo1:~]$ module load conda
[mandresm@albedo1:~]$ module load analysis-toolbox
[mandresm@albedo1:~]$ jupyter notebook --no-browser --ip=0.0.0.0
...
[I 15:26:36.310 NotebookApp] Jupyter Notebook 6.5.3 is running at:
[I 15:26:36.310 NotebookApp] http://albedo1:8891/?token=asdasdads
[I 15:26:36.310 NotebookApp]  or http://127.0.0.1:8891/?token=asdasdasd
[I 15:26:36.310 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:26:36.313 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///albedo/home/mandresm/.local/share/jupyter/runtime/nbserver-3890270-open.html
    Or copy and paste one of these URLs:
        http://albedo1:8891/?token=asdasdaas
     or http://127.0.0.1:8891/?token=asdasda

On the local machine, paste the URL including the albedo0 or albedo1 word into your browser, but replace albedo0 or albedo1 with albedo0.dmawi.de or albedo1.dmawi.de respectively. For the example above the address for the browser would be http://albedo1.dmawi.de:8891/?token=asdasdaas

JupyterLab from a COMPUTE or a GPU node

This example covers how to request a GPU node, but doing the same with a COMPUTE node would be almost identical, you will just need to remove the GPU parts on the salloc call. 

Background: By default, Jupyer notebook uses /run/user/<uid> as default directory for small files like notebook_cookie_secret. If you log in by ssh, /run/user/<uid> is created and it is removed when you close your last login session on the computer. However, if you enter a node via Slurm sbatch, salloc, or srun, /run/user/<uid> is not available. XDG_RUNTIME_DIR sets a different path.

mandresm@albedo1:~$ salloc --partition=gpu --gpus=1 -A computing.computing --time=00:30:00
salloc: Pending job allocation 6526219
salloc: job 6526219 queued and waiting for resources
salloc: job 6526219 has been allocated resources
salloc: Granted job allocation 6526219
salloc: Waiting for resource configuration
salloc: Nodes gpu-001 are ready for job


mandresm@gpu-001:~$ export XDG_RUNTIME_DIR="/tmp/tmp_$SLURM_JOBID"
mandresm@gpu-001:~$ module load conda
mandresm@gpu-001:~$ module load analysis-toolbox
mandresm@gpu-001:~$ jupyter notebook --no-browser --ip=0.0.0.0
...
[I 15:37:11.953 NotebookApp] Serving notebooks from local directory: /albedo/home/mandresm
[I 15:37:11.953 NotebookApp] Jupyter Notebook 6.5.3 is running at:
[I 15:37:11.953 NotebookApp] http://gpu-001:8888/?token=123
[I 15:37:11.953 NotebookApp]  or http://127.0.0.1:8888/?token=123
[I 15:37:11.953 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:37:11.958 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///albedo/home/mandresm/.local/share/jupyter/runtime/nbserver-698858-open.html
    Or copy and paste one of these URLs:
        http://gpu-001:8888/?token=123
     or http://127.0.0.1:8888/?token=123

Now, we have to establish an SSH tunnel from your PC to the compute node, in this case gpu-001, to forward the Jupyter Notebook. Check the port number - usually 8888 for jupyter notebook, but it might differ. Open a new local terminal and execute:

mandresm@blik0256:~$ ssh -NL localhost:8888:gpu-001:8888 mandresm@albedo1.dmawi.de

If you have ssh automatically configured to connect to Albedo then the process will be idling at this point. If you are requested your password then enter your password and again there will be no output. You don't need to do anything here anymore, just leave this local terminal open.

Now copy the address that looks like http://gpu-001:8888/?token=123 paste it in your browser, and substitute gpu-001 with 0.0.0.0. Jupyter Lab should open now!

How to use your own conda environment in JupyterHub/Lab? → add a Jupyter Kernel?

Python Notebook

For using your own conda environment in JupyterHub/Lab and access it via a Kernel there make sure your conda environment already has ipykernel:

conda activate <your_conda_environment>
conda install ipykernel

Then install your environment as an ipykernel:

conda activate {environment_name}
python -m ipykernel install --user --name=<environment_name>

Refresh Jupyter in your browser, you should now have your environment available as a kernel.

R Notebook

To create an R kernel for your own instance of Jupyter load the R module:

module load r

Then install the kernel with IRkernel, making sure you name it differently than just simply "R" (e.g. "my_R_kernel"):

Rscript -e 'IRkernel::installspec(name="<your_R_kernel>", displayname="<your_R_kernel")'

Refresh Jupyter in your browser, you should now have your environment available as a kernel. Whatever you had installed in your instance of R should be available via that kernel's notebooks.

jupyter-client has to be installed but “jupyter kernelspec --version” exited with code 1.

$ module load gcc
$ R
> install.packages("devtools")
> devtools::install_github("IRkernel/IRkernel")

Singularity

Singularity (now renamed Apptainer) is a containerization software similar to Docker but with several additional security features which make it feasible to use on HPC systems. You can find more information about the software here: https://apptainer.org/docs/user/main/

We offer Apptainer/Singularity as a module which can be loaded with:

$ module load apptainer

Thereafter, you should have both the singularity and apptainer executables available to you so you can download and run containers. These programs are interchangeable.

Important note about building containers from scratch: Building requires root privileges! However, the generated container files are portable and can be copied from (e.g.) your personal laptop to the HPC system. Alternatively, you can consider to use the "remote builder" hosted at Sylabs.io: https://cloud.sylabs.io/builder

Matlab

Currently the versions (R2022b) and (R2025b) of Matlab are available on albedo.

We offer the software as a module accessible by

module load matlab

After loading the module the program is started by

matlab

Please do not use Matlab on the login nodes, instead use the compute nodes fat-003 and fat-004 that are accessed by the slurm partition matlab.

Necessary ressources may be allocated by the slurm command initializing your Matlab session. Please adjust the example

salloc -p matlab --x11 --mem=8GB --time=2:00:00 --qos=12h --account=<user_slurm_account>

to your needs (and append your account if necessary). Since the nodes are shared between all Matlab users there are limitations implemented with respect to CPU and memory usage.

Please activate your personal license if you have one. This is done just like on any other platform by running

activate_matlab.sh

(after loading the module). A GUI will guide you through the necessary steps for activation of your personal license. Please note that you need to activate a license on each node you would like to use for Matlab (by using --nodelist=fat-003 or fat-004).

OpenCV

You might need before your script if you use OpenCV:

$ module load mesa

IDE/ENVI

We were reported that idede, the graphical Interface for IDL, might crash because it might require more virtual memory than we allow. The IDL support provided the following solution:

I have shared your case with the team and would like to share the additional information:
1) First of all, when using older IDL version such as IDL 8.6, you can try to start IDLDE with the below steps to try to workaround the issue:
To be sure everything works fine, please first delete your ".idl" folder which can be found in the user home directory: "/home/user/"
Then run your IDLDE session with the following command:   " idlde -outofprocess "
(This will separate the java process which is running in the background.)

2) The virtual memory is somewhat alarming, but the good thing is that we are sure it’s not actually using that much memory.
However, that can still cause problems, like you are currently experiencing.

First of all, can you please confirm that you're not using an IDL startup script that is pre-allocating a bunch of array space? 

In addition, we noticed that you are using a non-standard system and it might be due to the used kernel and glibc version.
Theoretically, given the right kernel and glibc version, IDL should be able to run.
But you might need to do some tweaking (like that environment variable below) to get IDL to run properly on this specific OS.
We have found a thread on the web, which says that it isn’t Eclipse but could be related to glibc: https://www.eclipse.org/forums/index.php/t/1082034/

They recommend setting some flag, MALLOC_ARENA_MAX=4.
We have never heard of that but it could be worth trying on this specific system.
Here is another thread that also mentions that same environment variable: https://stackoverflow.com/questions/561245/virtual-memory-usage-from-java-under-linux-too-much-memory-used

Panoply

Panoply plots geo-referenced and other arrays from netCDF, HDF, GRIB, and other datasets. To use its graphical interface make sure you login into albedo via ssh with X forwarding (ssh -X ...). Then run the following commands:

module load panoply
panoply.sh

uftp-client

uftp is a parallel data transfer tool that uses multiple streams to transfer large ammounts of data efficiently between different systems. The tool has two main components: a server and a client. Albedo does not have a uftp server, but other HPC centers do have one (e.g. DKRZ and Jülich), which means you can efficiently transfer data from and to Albedo to and from these other HPC centers, using Albedo's uftp-client. For that follow these steps:

1. Follow the steps to register an ssh-key in the target uftp-server, and read the full uftp-server documentation:
   
   1. For data transfers from/to Levante: https://docs.dkrz.de/doc/levante/data-transfer/uftp.html
   2. For data transfers from/to Jülich: https://apps.fz-juelich.de/jsc/hps/judac/uftp.html
2. Load the module on Albedo:
   

   module load uftp-client

3. Use the commands referred in the documentation (and uftp documentation) above to transfer your data.

OpenFOAM

OpenFOAM is installed on Albedo and available as a module:

$ module load avail
$ module load openfoam/<version>

To install an external OpenFOAM library/solver you'd need to load openfoam from spack instead, so that the environment and the dependencies are set up for you automatically:

$ module purge
$ module load spack
$ spack load openfoam/<version>

Then follow the installation instructions of the library you are trying to install. They can involve running a make file. In that case make sure you set up the necessary environment variables for building in a directory where you have writting access, otherwise, you'll end up with an error similar to:

mkdir: cannot create directory ‘’: No such file or directory                                                                                                                                                                    
make: *** [/albedo/soft/sw/spack-sw/openfoam/2112-u257d6d/wmake/makefiles/general:182: /libhydrology.so] Error 1

This is an example for how to solve this problem for the hydrology library:

$ export FOAM_USER_APPBIN=/path/with/writting/permissions/lib # I found about these variables in the make file Allwmake of the hydrology package
$ export FOAM_USER_APPBIN=/path/with/writting/permissions/bin
$ ./Allwmake

PyTorch

PyTorch is currently not installed in the cluster as a module or conda environment. If you'd like to install it for your own use (or for your group inside a data project) we recommend that you install it in a conda environment, to keep the environment isolated from other applications. What follows is a recommendation on how to install Torch within a conda environment at AWI's HPC:

1. In https://pytorch.org/ read the Install section and all the install documentation. You'll see a matrix such as the one below where you can check what prerequisites you need for the installation of the desired version of Torch.
   
   
2. Create an environment.yaml file to describe the conda environment you need. Note that installing Torch via the conda package manager is not supported anymore, so you'll need to use pip to install it within the conda environment. You can use the environment.yaml bellow as a template:
   
   environment.yaml

   name: pytorch
   channels:
   - conda-forge
   dependencies:
   - python>=3.12
   - pip 
   - pip: # This is what makes the installation via pip instead of via conda/mamba
       - torch
       - torchvision
       - torchaudio 

3. If you want to build it with GPU support start an interactive sessions in a GPU node, for example:
   

   salloc -A <account> --partition=gpu --gpus=a40:1

4. If you want to build it with GPU support, load the version of cuda compatible with the version of Torch you need, for example:

   module load cuda/12.4

5. Load conda and create the environment:
   

   module load conda
   mamba env create -f environment.yaml

6. Load the installed conda environment, check that you can import Torch and, if you need GPU support, check whether cuda is available to it:
   

   conda activate pytorch
   python
   Python 3.13.2 | packaged by conda-forge | (main, Feb 17 2025, 14:10:22) [GCC 13.3.0] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import torch
   >>> torch.cuda.is_available()
   True
   >>> print(torch.version.cuda)
   12.4

VSCode and other code editors

Although we recommend the use of JupyterHub for tasks requiring a GUI, you can also connect to Albedo using editors such as VSCode, Windsurf, RStudio, Cursor, or others that support remote SSH connections. The editor interface runs locally on your machine, while files and computations happen on Albedo’s servers. This reduces delay compared to running the GUIs directly on the server while using `ssh -X`/`ssh -Y`, and gives you powerful editing features.

In order to keep things simple we will address only VSCode in this section, but the same principles apply to other editors.

VSCode for Login Nodes

Login nodes are the entry point to Albedo. They allow you to access the system, submit jobs, install software, and run light tasks.

VSCode for computing nodes

# Login node configuration, same as in ssh-keys section. It does not need
# to look like this example, as long as it works for you to connect to the login node
Host albedo1
    HostName albedo1.dmawi.de
    User <your_username>
    IdentityFile ~/.ssh/id_ed25519_albedo

# This is the important part for connecting to compute nodes
Host prod-<XXX>
    HostName prod-<XXX>
    User <your_username>
    IdentityFile ~/.ssh/id_ed25519_production
    ProxyJump albedo1

salloc --account=<your_account> --partition=smp --mem=128G --cpus-per-task=16 --qos=12h --time=05:30:00

How to close VSCode to avoid problems

In the section below we explain how to trouble shoot some of the VSCode problems, but one way of avoiding some of those problems is to closing the connection so VSCode whenever you need to stop working with VSCode in Albedo. To do that follow this steps:

1. Instead of just closing the VSCode window (which might leave VSCode servers unnecesarily running in Albedo), click on the SSH: Albedo button at the bottom left corner of VSCode:
   
2. Then select "Close remote connection":
   

Trouble-shooting issues with VSCode

We have noticed that sometimes VSCode runs updates that open multiple logins in the login nodes. Other times VSCode keeps a server running in the login node when it should have close it, and it becomes an orphan process in the background that runs under your linux user. We have also seen that some workloads with VSCode might spawn more than one processes or logins. Maximum number of login sessions per user is 6 in Albedo login nodes, because of security reasons. If you reach that limit you'll see errors while trying to connect to Albedo through VSCode, such as:

If you look at the Output of the ssh connection you'll see "Too many logins for '<your_user>'"

If you are hitting this problem or similar, please try the following:

1. Close other existing connections to that login node via VSCode
2. If you have an terminal connected to ssh to that loging node, try to find how many `vscode-server` processes do you have running. If the number is high (e.g. more than 4) consier killing those processes: `pkill -u $USER -f '\.vscode-server' `
3. If you are not able to connect via terminal to ssh try running the following command remotely on the login node via SSH to list the \.vscode-server processes:

   1. ssh albedo0/1.dmawi.de ps -fu $USER | grep vscode-server

4. If the number of vscode-servers is high (e.g. mode than 4) consider killing those processes remotely:

   1. ssh lbedo0/1.dmawi.de pkill -u $USER -f '\.vscode-server'

5. If that does not work please let us know by contacting us at hpc@awi.de.

If you can connect to the login nodes but the the VSCode session is terminating with an error complaining about "Extensions", this might be because the extensions are using more resources than allowed in a login node. In that case, please consider narrowing down what extension is giving you the problem by deactvating one by one the extensions. You can also try connecting to a compute node instead while having allocated suggicient resources.