module avail

Compiler & MPI

We currently provide the following compilers:

Compiler options for optimization

Some remarks from Natalja with respect to the intel compiler and the FESOM2 benchmark (used by NEC in there albedo offer):

1. do not  use -xHost, because Intel does not "recognize" AMD (officially for security reasons . Therefore use: -xcore-avx2
2. These compiler options were used by NEC during the FESOM2 benchmark:

   OPT = -O3 -qopt-report5 -no-prec-div -ip -fp-model=fast=2 -implicitnone -march=core-avx2 -fPIC –qopenmp -qopt-malloc-options=2 -qopt-prefetch=5 -unroll-aggressive

   These are (at least partially) quite important for good performance. However, we do not have the experience which are more or less critical. Be careful, some options might kill reproducibility (e.g., -fp-model=fast=2).
3. Natalja is now responsible for this: https://docs.dkrz.de/doc/levante/running-jobs/runtime-settings.html#open-mpi-4-0-0-and-lat let's stll benefit from here knowledge
4. Independent on the MPI used, please try runtime setting
   UCX_TLS=knem,dc_x,self
for your jobs. According to NEC for smaller Jobs it might be beneficial to replace "dc_x" with "rc_x".
5. When using OpenMPI, parallel IO (using OMPIO) might be slow. Try using
   

   export OMPI_MCA_io="romio321"

Spack

On albedo we mainly use spack to install software and provide module files.

On albedo it can also be used by users to

• install their own software into their $HOME
• load specific packages (similarly to what environment modules do)

Simply load the spack module:

module load spack

This version is configured, such that it makes use of the global software tree, installed by the admins, and your installations go into $HOME/.spack .

Please consult the official documentation on how to use spack: https://spack.readthedocs.io

Python, R, conda, Jupyter

Python

We provide Python modules with only basic Python installed for currently supported Python versions 3.7 through 3.10. Further packages you might need on top of this can be installed via standard methods, e.g. pip.

Additionally, we provide a toolbox of common data analysis tools in both Python and R. This module is available under analysis_toolbox and is updated every 3 months. This module is actually a conda environment which simply sets the correct shell variables (e.g. $PATH, $PYTHONPATH, and similar) for you. A list of currently installed Python and R tools in this module may be found under /albedo/soft/conda-workspace/env-ymls/analysis-toolbox-03.2023.yml.

If you would like additional libraries in this globally available analysis toolbox, just ask! They will be added whenever the next one is released (January, March, June, and September)

R

Similar to Python, the some basic R modules are included in the analysis_toolbox. Currently, there is no R-Studio available but we are planning to install it in the September-2023 update of the analysis_toolbox. Be aware that we still recommend against using graphical interfaces since that's not what an HPC is designed for. Our recommended workflow for using R is either with the interactive session and the R, or via Rscript and sbatch script:

Interactive session with R command line interface

From Albedo's login node:

salloc --account=<your_account> --time=<HH:MM:SS> --qos=<QOS> --nodes=<#Nodes> <other_options...>

To understand which options you need to specify please refer to the Jobs section on Albedo-Slurm and the SLURM user guide.

module load analysis-toolbox
R

The R command line will open and you can start using R from there.

Rscript and sbatch script

From Albedo's login node:

$ module load analysis-toolbox

The command Rscript allows you to run an R script you wrote from the shell (outside of R IDE or R-Studio). This means that you can also write a sbatch script (see Albedo-Slurm > Jobs) that runs your R scripts via Rscript and submit it to the slurm queue using sbatch command. You can even use the slurm array feature to launch the same R script multiple times in parallel if what your script does can be broken into multiple computing chunks. Depending on your R script it might need more or less changes, but it's probably worth to spend the time on chaning it to be able to benefit from this first order parallelization. For a simple example on slurm array + Rscript the following tutorial covers most of it: https://rcpedia.stanford.edu/topicGuides/jobArrayRExample.html

Conda

Conda is a package manager for Python, R, and Julia software. You can use it on our HPC system by:

$ module load conda

Thereafter, you should be able to use conda to manage your Python/R/Julia environments. Typically you will want to install software via:

$ conda install -c conda-forge <PACKAGE_NAME>

Note that this allows you to install both Python as well as R packages. Full documentation for conda is provided here: https://docs.conda.io/en/latest/ 

A useful cheat sheet for using conda can be found here: https://docs.conda.io/projects/conda/en/4.6.0/_downloads/52a95608c49671267e40c689e0bc00ca/conda-cheatsheet.pdf

Jupyter

Jupyter is an interactive computing environment which lets you execute notebooks which can mix code, text, graphics, and LaTeX all in a single document. There are different ways to use Jupyter from Albedo listed bellow.

JupyterHub

See Jupyterhub on Albedo.

JupyterLab from a login node

Please, read until the end of this section, the last step is really important for things to work. Load the analysis-toolbox:

[mandresm@albedo1:~]$ module load conda
[mandresm@albedo1:~]$ module load analysis-toolbox
[mandresm@albedo1:~]$ jupyter notebook --no-browser --ip=0.0.0.0
...
[I 15:26:36.310 NotebookApp] Jupyter Notebook 6.5.3 is running at:
[I 15:26:36.310 NotebookApp] http://albedo1:8891/?token=asdasdads
[I 15:26:36.310 NotebookApp]  or http://127.0.0.1:8891/?token=asdasdasd
[I 15:26:36.310 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:26:36.313 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///albedo/home/mandresm/.local/share/jupyter/runtime/nbserver-3890270-open.html
    Or copy and paste one of these URLs:
        http://albedo1:8891/?token=asdasdaas
     or http://127.0.0.1:8891/?token=asdasda

On the local machine, paste the URL including the albedo0 or albedo1 word into your browser, but replace albedo0 or albedo1 with albedo0.dmawi.de or albedo1.dmawi.de respectively. For the example above the address for the browser would be http://albedo1.dmawi.de:8891/?token=asdasdaas

JupyterLab from a COMPUTE or a GPU node

This example covers how to request a GPU node, but doing the same with a COMPUTE node would be almost identical, you will just need to remove the GPU parts on the salloc call. 

Background: By default, Jupyer notebook uses /run/user/<uid> as default directory for small files like notebook_cookie_secret. If you log in by ssh, /run/user/<uid> is created and it is removed when you close your last login session on the computer. However, if you enter a node via Slurm sbatch, salloc, or srun, /run/user/<uid> is not available. XDG_RUNTIME_DIR sets a different path.

mandresm@albedo1:~$ salloc --partition=gpu --gpus=1 -A computing.computing --time=00:30:00
salloc: Pending job allocation 6526219
salloc: job 6526219 queued and waiting for resources
salloc: job 6526219 has been allocated resources
salloc: Granted job allocation 6526219
salloc: Waiting for resource configuration
salloc: Nodes gpu-001 are ready for job


mandresm@gpu-001:~$ export XDG_RUNTIME_DIR="/tmp/tmp_$SLURM_JOBID"
mandresm@gpu-001:~$ module load conda
mandresm@gpu-001:~$ module load analysis-toolbox
mandresm@gpu-001:~$ jupyter notebook --no-browser --ip=0.0.0.0
...
[I 15:37:11.953 NotebookApp] Serving notebooks from local directory: /albedo/home/mandresm
[I 15:37:11.953 NotebookApp] Jupyter Notebook 6.5.3 is running at:
[I 15:37:11.953 NotebookApp] http://gpu-001:8888/?token=123
[I 15:37:11.953 NotebookApp]  or http://127.0.0.1:8888/?token=123
[I 15:37:11.953 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:37:11.958 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///albedo/home/mandresm/.local/share/jupyter/runtime/nbserver-698858-open.html
    Or copy and paste one of these URLs:
        http://gpu-001:8888/?token=123
     or http://127.0.0.1:8888/?token=123

Now, we have to establish an SSH tunnel from your PC to the compute node, in this case gpu-001, to forward the Jupyter Notebook. Check the port number - usually 8888 for jupyter notebook, but it might differ. Open a new local terminal and execute:

mandresm@blik0256:~$ ssh -NL localhost:8888:gpu-001:8888 mandresm@albedo1.dmawi.de

If you have ssh automatically configured to connect to Albedo then the process will be idling at this point. If you are requested your password then enter your password and again there will be no output. You don't need to do anything here anymore, just leave this local terminal open.

Now copy the address that looks like http://gpu-001:8888/?token=123 paste it in your browser, and substitute gpu-001 with 0.0.0.0. Jupyter Lab should open now!

How to use your own conda environment in JupyterHub/Lab? → add a Jupyter Kernel?

Python Notebook

For using your own conda environment in JupyterHub/Lab and access it via a Kernel there make sure your conda environment already has ipykernel:

conda activate <your_conda_environment>
conda install ipykernel

Then install your environment as an ipykernel:

conda activate {environment_name}
python -m ipykernel install --user --name=<environment_name>

Refresh Jupyter in your browser, you should now have your environment available as a kernel.

R Notebook

To create an R kernel for your own instance of Jupyter load the R module:

module load r

Then install the kernel with IRkernel, making sure you name it differently than just simply "R" (e.g. "my_R_kernel"):

Rscript -e 'IRkernel::installspec(name="<your_R_kernel>", displayname="<your_R_kernel")'

Refresh Jupyter in your browser, you should now have your environment available as a kernel. Whatever you had installed in your instance of R should be available via that kernel's notebooks.

Singularity

Singularity (now renamed Apptainer) is a containerization software similar to Docker but with several additional security features which make it feasible to use on HPC systems. You can find more information about the software here: https://apptainer.org/docs/user/main/

We offer Apptainer/Singularity as a module which can be loaded with:

$ module load apptainer

Thereafter, you should have both the singularity and apptainer executables available to you so you can download and run containers. These programs are interchangeable.

Important note about building containers from scratch: Building requires root privileges! However, the generated container files are portable and can be copied from (e.g.) your personal laptop to the HPC system. Alternatively, you can consider to use the "remote builder" hosted at Sylabs.io: https://cloud.sylabs.io/builder

Matlab

Currently the version (R2022b) of Matlab is available on albedo.

We offer the software as a module accessible by

module load matlab

After loading the module the program is started by

matlab

Right now the usage on compute nodes is still being set up, currently, please use Matlab on the compute nodes fat-003 and fat-004 accessed by the slurm partition matlab.

Necessary ressources may be allocated by the slurm command initializing your Matlab session. Please adjust the example

salloc -p matlab --x11 --mem=8GB --time=2:00:00 --qos=12h

to your needs (and append your account if necessary). Since the nodes are shared between all Matlab users there are limitations implemented with respect to CPU and memory usage.

Please activate your personal license if you have one. This is done just like on any other platform by running

activate_matlab.sh

(after loading the module). A GUI will guide you through the necessary steps for activation of your personal license. Please note that you need to activate a license on each node you would like to use for Matlab (by using --nodelist=fat-003 or fat-004).

OpenCV

You might need before your script if you use OpenCV:

$ module load mesa

IDE/ENVI

We were reported that idede, the graphical Interface for IDL, might crash because it might require more virtual memory than we allow. The IDL support provided the following solution:

I have shared your case with the team and would like to share the additional information:
1) First of all, when using older IDL version such as IDL 8.6, you can try to start IDLDE with the below steps to try to workaround the issue:
To be sure everything works fine, please first delete your ".idl" folder which can be found in the user home directory: "/home/user/"
Then run your IDLDE session with the following command:   " idlde -outofprocess "
(This will separate the java process which is running in the background.)

2) The virtual memory is somewhat alarming, but the good thing is that we are sure it’s not actually using that much memory.
However, that can still cause problems, like you are currently experiencing.

First of all, can you please confirm that you're not using an IDL startup script that is pre-allocating a bunch of array space? 

In addition, we noticed that you are using a non-standard system and it might be due to the used kernel and glibc version.
Theoretically, given the right kernel and glibc version, IDL should be able to run.
But you might need to do some tweaking (like that environment variable below) to get IDL to run properly on this specific OS.
We have found a thread on the web, which says that it isn’t Eclipse but could be related to glibc: https://www.eclipse.org/forums/index.php/t/1082034/

They recommend setting some flag, MALLOC_ARENA_MAX=4.
We have never heard of that but it could be worth trying on this specific system.
Here is another thread that also mentions that same environment variable: https://stackoverflow.com/questions/561245/virtual-memory-usage-from-java-under-linux-too-much-memory-used

Panoply

Panoply plots geo-referenced and other arrays from netCDF, HDF, GRIB, and other datasets. To use its graphical interface make sure you login into albedo via ssh with X forwarding (ssh -X ...). Then run the following commands:

module load panoply
panoply.sh

uftp-client

uftp is a parallel data transfer tool that uses multiple streams to transfer large ammounts of data efficiently between different systems. The tool has two main components: a server and a client. Albedo does not have a uftp server, but other HPC centers do have one (e.g. DKRZ and Jülich), which means you can efficiently transfer data from and to Albedo to and from these other HPC centers, using Albedo's uftp-client. For that follow these steps:

1. Follow the steps to register an ssh-key in the target uftp-server, and read the full uftp-server documentation:
   
   1. For data transfers from/to Levante: https://docs.dkrz.de/doc/levante/data-transfer/uftp.html
   2. For data transfers from/to Jülich: https://apps.fz-juelich.de/jsc/hps/judac/uftp.html
2. Load the module on Albedo:
   

   module load uftp-client

3. Use the commands referred in the documentation (and uftp documentation) above to transfer your data.

OpenFOAM

OpenFOAM is installed on Albedo and available as a module:

$ module load avail
$ module load openfoam/<version>

To install an external OpenFOAM library/solver you'd need to load openfoam from spack instead, so that the environment and the dependencies are set up for you automatically:

$ module purge
$ module load spack
$ spack load openfoam/<version>

Then follow the installation instructions of the library you are trying to install. They can involve running a make file. In that case make sure you set up the necessary environment variables for building in a directory where you have writting access, otherwise, you'll end up with an error similar to:

mkdir: cannot create directory ‘’: No such file or directory                                                                                                                                                                    
make: *** [/albedo/soft/sw/spack-sw/openfoam/2112-u257d6d/wmake/makefiles/general:182: /libhydrology.so] Error 1

This is an example for how to solve this problem for the hydrology library:

$ export FOAM_USER_APPBIN=/path/with/writting/permissions/lib # I found about these variables in the make file Allwmake of the hydrology package
$ export FOAM_USER_APPBIN=/path/with/writting/permissions/bin
$ ./Allwmake