Login nodes
The two login nodes are the only accessible nodes from the AWI intranet.
Quantity | Name | Specification |
|---|---|---|
2x | albedo[0|1] |
|
Compute nodes
Quantity | Name | Partition | Specification | Notes |
|---|---|---|---|---|
240x | prod-[001-240] | smp, smpht |
| For our test phase, we have split the compute nodes in two sets More information can be found on the slurm documentation. |
4x | fat-00[1-4] | fat, matlab |
| fat-00[3,4] are currently reserved for matlab users, this might change later |
1x | gpu-001 | gpu |
| A comparison of the two different GPUs can be found here:
|
1x | gpu-002 | gpu |
|
Filesystem
Local user storage
- The local storage is a parallel GxFS Storage Appliance from NEC based on IBM Spectrum Scale: https://en.wikipedia.org/wiki/GPFS.
Tier 1 "system": ~171 TiB NVMe as fast cache and/or burst buffer
Tier 2 "data": ~5030 TiB NL-SAS HDD (NetApp EF300)
- You can check in which storage pool your data resides with mmlsattr -L <file>
- All nodes are connected via a 100 Gb Mellanox/Inifiniband network.
| Personal directories | Project directories | |||||
|---|---|---|---|---|---|---|
| Mountpoint | /albedo/home/$USER | /albedo/work/user/$USER | /albedo/scratch/user/$USER | /albedo/work/projects/$PROJECT | /albedo/scratch/projects/$PROJECT | /albedo/burst |
| Comes with | HPC_user account: https://id.awi.de → Start a new request/Bestellung → IT Service → HPC → Add to chart/In den Einkaufswagen | Apply for Quota here: https://cloud.awi.de/#/projects | -- | |||
| Block Quota | 100 GB (fixed) | 3 TB (fixed) | 50 TB (fixed) | 30 €/TB/yr (variable) | 10 €/TB/yr (variable) | |
| File Quota | 1e6 | 1e6 | max(1,log(1.5*BlockQuota)) * 1e6 | |||
| Delete | 90 days after user account expired | all data older than 90 days | 90 days after project expired | all data older then 90 days | after 10 days | |
| Security | Snapshots for 100 days | -- | Snapshots for 100 days | -- | -- | |
| Snapshots | /albedo/home/.snapshots/ | /albedo/work/user/.snapshots/ | -- | /albedo/work/projects/.snapshots/ | -- | |
| Owner:Group | $USER:hpc_user | $OWNER:$PROJECT | root:root | |||
| Permissions | 2700 → drwx--S--- | 2770 → rwxrws--- | 1777 → rwxwrxrwt | |||
| Focus | many small files | large files, large bandwidth | large files, large bandwidth | low latency, huge bandwidth | ||
Storage Pools
If you want to share data with other users use project directories. For convenience, a project administrator may request a link from /albedo/work/projects/$PROJECT/<somewhere> → /albedo/pool/<something> via hpc@awi.de
Remote user storage (/isibhv)
- You can access your online space on the Isilon in Bremerhaven (see https://spaces.awi.de/x/a13-Eg for more information) via the nfs-mountpoints
/isibhv/projects
/isibhv/projects-noreplica
/isibhv/netscratch
/isibhv/platforms
/isibhv/home - Tape storage (HSM) is not mounted. However, you could archive your results with something like
rsync -Pauv /albedo/work/projects/$PROJECT/my_valuable_results/* hssrv1:/hs/projects/$PROJECT/my_valuable_results_from_albedo/
Considerations: Where should I store my data?
| /work | Subject | /isibhv | ||
|---|---|---|---|---|
| pro | con | pro | con | |
| 100 Gb Infiniband | albedo intern only | Acceptability | available from everywhere (inside AWI) | 10 Gb Ethernet |
| low | Latency | higher | ||
| about 10-30 €/TB/yr | Cost | about 100-125 €/TB/yr | ||
| snapshots | Security | snapshots, automatic tape backup available (+25 €/TB/yr) | ||
Network
Fast interconnect (beween albedo's nodes):
HDR Infiniband
Ethernet:
- albedo is connected to the AWI backbone (including the Isilon and the HSM) via four eth-100 Gb interfaces.
Each single albedo node has a 10 Gb interface.