...

From Albedo's login node:

...

salloc --account=<your_account> --time=<HH:MM:SS> --qos=<QOS> --nodes=<#Nodes> <other_options...>

To understand which options you need to specify please refer to the the Jobs section on Albedo-Slurm and the SLURM user guide.

...

module load analysis-toolbox

...

R

The R command line will open and you can start using R from there.

...

Jupyter is an interactive computing environment which lets you execute notebooks which can mix code, text, graphics, and LaTeX all in a single document. There are different ways to use Jupyter from Albedo listed bellow.

JupyterHub

...

We are currently working on an experimental JupyterHub which will allow you to log on to the login nodes and run notebooks directly from the browser. If you want to test this out, you can try here: http://albedo0.dmawi.de:8000 Note that VPN is required! Currently access is provided only by request, please open a ticket on hpc@awi.de to get on the list during the testing phase.

You will be presented with a login page, and after login, with a selection of job profiles. You can either run your notebook on a login node (not recommended), a compute node, or a GPU node. In case of a compute node or GPU node, yu need to specify which computing account SLURM should use. A list of available computing accounts is provided for you. Additionally, in the case of a GPU node, you need to specify which type of GPU you want to use (A40 or A100) and how many GPUs you wish to use. 

...

See Jupyterhub on Albedo.

JupyterLab from a login node

Please, read until the end of this section, the last step is really important for things to work. Load the analysis-toolbox:

[mandresm@albedo1:~]$ module load conda
[mandresm@albedo1:~]$ module load analysis-toolbox
[mandresm@albedo1:~]$ jupyter notebook --no-browser --ip=0.0.0.0
...
[I 15:26:36.310 NotebookApp] Jupyter Notebook 6.5.3 is running at:
[I 15:26:36.310 NotebookApp] http://albedo1:8891/?token=asdasdads
[I 15:26:36.310 NotebookApp]  or http://127.0.0.1:8891/?token=asdasdasd
[I 15:26:36.310 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:26:36.313 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///albedo/home/mandresm/.local/share/jupyter/runtime/nbserver-3890270-open.html
    Or copy and paste one of these URLs:
        http://albedo1:8891/?token=asdasdaas
     or http://127.

$ jupyter kernelspec install /albedo/soft/sw/conda-sw/analysis-toolbox/03.2023/share/jupyter/kernels/python3 --name "analysis-toolbox_03.2023" 

Ensure you replace the path and name with appropriate values for your path! In your conda environment, you need to have the ipykernel package installed.  

SLURM-enabled Jupyterhub jobs are restricted to 12 hours.

JupyterLab from a login node

Load the analysis-toolbox:

[mandresm@albedo1:~]$ module load analysis-toolbox
[mandresm@albedo1:~]$ jupyter notebook --no-browser --ip=0.0.0.0
...
[I 15:26:36.310 NotebookApp] Jupyter Notebook 6.5.3 is running at:
[I 15:26:36.310 NotebookApp] http://albedo1:8891/?token=asdasdads
[I 15:26:36.310 NotebookApp]  or http://127.0.0.1:8891/?token=asdasdasd
[I 15:26:36.310 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:26:36.313 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///albedo/home/mandresm/.local/share/jupyter/runtime/nbserver-3890270-open.html
    Or copy and paste one of these URLs:
        http://albedo1:8891/?token=asdasdaas
     or http://127.0.0.1:8891/?token=asdasda

On the local machine, paste the URL including the albedo0 or albedo1 word into your browser, but replace albedo0 or albedo1 by albedo0.dmawi.de or albedo1.dmawi.de respectively.

JupyterLab from a COMPUTE or a GPU node

This example covers how to request a GPU node, but doing the same with a COMPUTE node would be almost identical, you will just need to remove the GPU parts on the salloc call. 

Background: By default, Jupyer notebook uses /run/user/<uid> as default directory for small files like notebook_cookie_secret. If you log in by ssh, /run/user/<uid> is created and it is removed when you close your last login session on the computer. However, if you enter a node via Slurm sbatch, salloc, or srun, /run/user/<uid> is not available. XDG_RUNTIME_DIR sets a different path.

asdasda

On the local machine, paste the URL including the albedo0 or albedo1 word into your browser, but replace albedo0 or albedo1 with albedo0.dmawi.de or albedo1.dmawi.de respectively. For the example above the address for the browser would be http://albedo1.dmawi.de:8891/?token=asdasdaas

JupyterLab from a COMPUTE or a GPU node

This example covers how to request a GPU node, but doing the same with a COMPUTE node would be almost identical, you will just need to remove the GPU parts on the salloc call. 

Background: By default, Jupyer notebook uses /run/user/<uid> as default directory for small files like notebook_cookie_secret. If you log in by ssh, /run/user/<uid> is created and it is removed when you close your last login session on the computer. However, if you enter a node via Slurm sbatch, salloc, or srun, /run/user/<uid> is not available. XDG_RUNTIME_DIR sets a different path.

mandresm@albedo1:~$ salloc --partition=gpu --gpus=1 -A computing.computing --time=00:30:00
salloc: Pending job allocation 6526219
salloc: job 6526219 queued and waiting for resources
salloc: job 6526219 has been allocated resources
salloc: Granted job allocation 6526219
salloc: Waiting for resource configuration
salloc: Nodes gpu-001 are ready for job


mandresm@gpu-001:~$ export XDG_RUNTIME_DIR="/tmp/tmp_$SLURM_JOBID"
mandresm@gpu-001:~$ module load conda
mandresm@gpu-001:~$ module load analysis-toolbox
mandresm@gpu-001:~$ jupyter notebook --no-browser --ip=0.0.0.0
...
[I 15:37:11.953 NotebookApp] Serving notebooks from local directory: /albedo/home/mandresm
[I 15:37:11.953 NotebookApp] Jupyter Notebook 6.5.3 is running at:
[I 15:37:11.953 NotebookApp] http://gpu-001:8888/?token=123
[I 15:37:11.953 NotebookApp]  or http://127.0.0.1:8888/?token=123
[I 15:37:11.953 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:37:11.958 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///albedo/home/mandresm/.local/share/jupyter/runtime/nbserver-698858-open.html
    Or copy and paste one of these URLs:
       

mandresm@albedo1:~$ salloc --partition=gpu --gpus=1 -A computing.computing --time=00:30:00
salloc: Pending job allocation 6526219
salloc: job 6526219 queued and waiting for resources
salloc: job 6526219 has been allocated resources
salloc: Granted job allocation 6526219
salloc: Waiting for resource configuration
salloc: Nodes gpu-001 are ready for job


mandresm@gpu-001:~$ export XDG_RUNTIME_DIR="/tmp/tmp_$SLURM_JOBID"
mandresm@gpu-001:~$ module load analysis-toolbox
mandresm@gpu-001:~$ jupyter notebook --no-browser --ip=0.0.0.0
...
[I 15:37:11.953 NotebookApp] Serving notebooks from local directory: /albedo/home/mandresm
[I 15:37:11.953 NotebookApp] Jupyter Notebook 6.5.3 is running at:
[I 15:37:11.953 NotebookApp] http://gpu-001:8888/?token=123
[I 15:37:11.953 NotebookApp]      or http://127.0.0.1:8888/?token=123
[I 15:37:11.953 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:37:11.958 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///albedo/home/mandresm/.local/share/jupyter/runtime/nbserver-698858-open.html
    Or copy and paste one of these URLs:
        http://gpu-001:8888/?token=123
     or http://127.0.0.1:8888/?token=123

Now, we have to establish an SSH tunnel from your PC to the compute node, in this case gpu-001, to forward the Jupyter Notebook. Check the port number - usually 8888 for jupyter notebook, but it might differ. Open a new local terminal and execute:

mandresm@blik0256:~$ ssh -NL localhost:8888:gpu-001:8888 mandresm@albedo1.dmawi.de

If you have ssh automatically configured to connect to Albedo then the process will be idling at this point. If you are requested your password then enter your password and again there will be no output. You don't need to do anything here anymore, just leave this local terminal open.

Now, we have to establish an SSH tunnel from your PC to the compute node, in this case gpu-001, to forward the Jupyter Notebook. Check the port number - usually 8888 for jupyter notebook, but it might differ. Open a new local terminal and execute:

mandresm@blik0256:~$ ssh -NL localhost:8888:gpu-001:8888 mandresm@albedo1.dmawi.de

If you have ssh automatically configured to connect to Albedo then the process will be idling at this point. If you are requested your password then enter your password and again there will be no output. You don't need to do anything here anymore, just leave this local terminal open.

Now copy the address that looks like http://gpu-001:8888/?token=123 paste it in your browser, and substitute gpu-001 with 0.0.0.0. Jupyter Lab should open now!

How to use your own conda environment in JupyterHub/Lab? → add a Jupyter Kernel?

Python Notebook

For using your own conda environment in JupyterHub/Lab and access it via a Kernel there make sure your conda environment already has ipykernel:

conda activate <your_conda_environment>
conda install ipykernel

Then install your environment as an ipykernel:

conda activate {environment_name}
python -m ipykernel install --user --name=<environment_name>

Refresh Jupyter in your browser, you should now have your environment available as a kernel.

R Notebook

To create an R kernel for your own instance of Jupyter load the R module:

module load r

Then install the kernel with IRkernel, making sure you name it differently than just simply "R" (e.g. "my_R_kernel"):

Rscript -e 'IRkernel::installspec(name="<your_R_kernel>", displayname="<your_R_kernel")'

Refresh Jupyter in your browser, you should now have your environment available as a kernel. Whatever you had installed in your instance of R should be available via that kernel's notebooks.Now copy the address that looks like http://gpu-001:8888/?token=123 pate it in your browser, and substitute gpu-001 with 0.0.0.0. Jupyter Lab should open now!

Singularity

...