Introduction
At AWI's HPC albedo, we offer a Jupyterhub service. You can log on under http://albedo0.dmawi.de:8000. Upon logging in, you are presented with a series of options; which will spawn a Jupyterlab Interface, similar to the single-user Lab. The jobs spawn on SLURM, and depending on the resources you select, have a maximum wall time ranging from 12 hours to 3 hours. If you want to use GPU resources, you also need to specify how many GPUs you would like to use.
Available Environments
By default, you are given several Conda-based environments which you can use for both Python and R. These are curated by the HPC team. Additionally, any environments you have produced yourself will be available to you. If you want to install your own kernel, please follow the instructions in Add a Jupyter Kernel
The Jupyter Interface
As you may already know Jupyter provides an interface for exploratory interaction with your data on compute resources. You get two interfaces by default; one is the new jupyterlab, and other is the classic notebook. You can freely switch between them Our jupyterhub enables you to use HPC resources for this exploration. We would however like to stress that this is only a supplement to more traditional HPC workflows, where you create batch jobs. Our team is happy to support you in finding an ideal workflow for your project! Our current approach recommendation would be the following:
- Use a notebook interface to figure out exactly what you are trying to determine from your data; on a sub-set of your data.
- Convert this notebook to a "pure
Python" script. - Generate a batch job script which runs this Python script to process all of your data.
At the moment, we support jobs up to 12 hours; this might be less depending on which resources your request (GPUs get less time)
SLURM Considerations
When using Jupyterhub, you are effectively running slurm jobs. The standard out and standard error (stdout & stderr) files are placed in your ${HOME} directory.
Which nodes am I using?
When using the jupyterhub interface, after logging in you are given some options, as mentioned above. Below the submit button, you will see how many resources are available for your particular job. These numbers dynamically update every second. Note that the GPU resources can be shared, so even if no "idle" node is available, you may still be able to use our GPUs. Information regarding GPU resources will be shown in more detail during the spawning page in the future :-)
Collaborative Editing
Another upcoming feature which is not yet extensively tested is the ability to collaboratively work on the same notebook together with colleagues. We will give a separate demo on this once we have gathered some experience. If you want to test this out; just let us know!
Updates to the Jupyterhub @ HPC
Occasionally, the admins will need to update the interface to enable new features. We will do our best to warn you beforehand, but particularly in the first few weeks, there may be unannounced down-times. We apologise for the inconvenience!