You can use this guide to get started using the Python programming language on AWI's HPC platform. For general help with Python, you can contact Dr. Paul Gierz (pgierz@awi.de) at the computing department.
Step-by-step guide
There are several ways to use Python on AWI's HPC platforms. This guide covers a traditional approach where you use a terminal, create unique environments for each project and use a standard terminal-based text editor. For other use cases, such as using Jupyter for interactive computing, or using a containerised version of Python, please see the separate guides. In this guide we cover setting up a conda environment, as well as activation and deactivation of environments.
Syntax Notation
In the following step-by-step guide, several commands will only need to be entered one single time. These are marked in a light blue info box. Others which you may need to do repeatedly, are not marked.
Code and terminal is prefixed with a $ symbol when using a standard shell, and >>> when using Python. Note that you should enter everything following the $, but not include it. Any other text might be the result of a particular command.
To begin, you will want to log on to AWI's HPC platform. Here I use the "Account for Basic Tests, HPC", user name abthpc, but you should of course substitute your own user name.
$ ssh -Y abthpc@ollie0.awi.de
The -Y flag for the ssh command enables window forwarding, which may be useful if you want to examine plots.
Next, you should examine your module environment. This specifies which software you currently have enabled, and as the name suggests, can be modularly changed depending on your current needs. We will then load miniconda, a small version of conda, a Python package manager.
$ module list No Modulefiles Currently Loaded. $ module load miniconda Module for miniconda version 4.10.3 loaded
Much like the module command modifies the software available to you to use for the entire computer, the conda command manipulates the Python version and accompanying libraries, called modules in Python terminology (not to be confused with the module command), you have available to you. Common Python modules you might encounter enable matrix computation, plotting, statistics, machine learning, parallel computing, and much more. A brief summary is given at the end of this article.
To see which conda environments are available, you can use the command
$ conda env list # conda environments: # base * /global/AWIsoft/miniconda/4.10.3 scicomp_dask_example /global/AWIsoft/miniconda/4.10.3/envs/scicomp_dask_example scicomp_esm_plot_notebook /global/AWIsoft/miniconda/4.10.3/envs/scicomp_esm_plot_notebook
You will see the base environment, and two example environments used for common demonstrations.
Ensure your shell is able to communicate with the conda environments. This is required as certain things like the shell $PATH are modified for each separate conda environment.
One Time Step
The following step only needs to be done once, and sets up your shell to interact with conda.
$ conda init bash
If you ever want to undo this, you can use:
$ conda init --reverse
More information can be easily retrieved with
$ conda init --help
You will now need to log out and back in again!
You can now activate an environment, for example:
$ conda activate scicomp_dask_example
This changes the Python version being used to the one that is in this particular conda environment. It will have access to Python modules installed in that environment.
You can create your own environment like this:
$ conda create -n <NAME_OF_ENVIRONMENT> python matplotlib numpy cartopy netcdf4 xarray