Slides: https://1drv.ms/p/c/9d02a8e5a472db02/Eb_uoDlcbyhDre5pFfJi9DUBVI-Qu3f4YMAFGtuLyDeTgw

Overview

Objective

The purpose of this meeting is to gather information on the current use of our local HPC system, Albedo, and to discuss options for the future development of AWI’s on-site HPC infrastructure once Albedo reaches the end of its operational lifetime.

AWI-HPC Forum

The AWI-HPC Forum is aimed at sections and groups that currently use, or plan to use, the local HPC resources at AWI.

Potential discussions

  • What do you like/dislike about the current system (Albedo)?
  • What are your HPC needs for the future?
  • What advantages/disadvantages/risks do you see in an HPC hosted locally at AWI?
  • What advantages/disadvantages/risks do you see in an AWI-dedicated HPC hosted at DKRZ?

List of Organizers

  • Stephan Frickenhaus (SF)
  • Miguel Andres-Martinez (MAM)
  • Sven Harig 
  • Ehsan Mehdipour

List of Participants

  • Wolfgang Dorn (WD)
  • Lars Nerger (LN)
  • Özgür Gürses (OG)
  • Veit Helm (VH)
  • Christina Bienhold (CB)
  • Malte Thoma (MT)
  • Daniel Feldmann (DF)
  • Mathias Campos van Caspel (MCvC)
  • Christian Stepanek 
  • Jonas Küpper (JK)
  • Frank Kauker (FK)
  • Finn Heukamp
  • Ralph Timmermann (RT)
  • Claudia Wekerle (CW)
  • Ingmar Nitze (IN)
  • Piers Larkman
  • Martin Losch (ML)
  • Martin Werner (MWe)
  • Jan Nitzbon (JN)
  • Raphael Köhler (RK)
  • Christoph Völker (CV)
  • Thomas Kleiner (TK)
  • Laetitia Roach
  • Matthias Wietz (MWi)
  • Thomas Jung


Protocol

Presentation

Introduction into background and aims by MAM and SF

Outline and presentation by MAM

  • List of options for hosting the future HPC system
  • Albedo overview
  • FTEs 
  • Current status: compute usage by sections/users
  • Current status: storage - parallel file system
  • Selected survey results

Discussion 

Q: What do you like/dislike about the current system (Albedo)?


Marine Bio Geo

  • OG: Advantage: easy to use, easy access. More problems with the archive. It is difficult to teach the use of the tape system, and opened a channel for that purpose.

Deep Sea ecology 

  • CB & MWi: Advantage: direct support
    Negative: Transfer data from external systems is tedious at times, and might be simplified by guest accounts

Glaciology

  • TK: Positive: Easy access, more downtimes on other systems (Lichtenberg?)
    • Negative: File system unpredictable performance degradation events
  • VH: Rely on data provision transfer to long-term storage. Large data handling is especially important. Dependence on Matlab/IDL/GAMMA. Hard to imagine transferring all the data of airborne campaigns and satellite remote sensing to external 

Permafrost

  • JN: Modeling, Remote sensing, satisfied as it is, some parallel structures wrt Potsdam. Better integration could help to get more users from Potsdam
  • IN: Transfer of data from P to BHV could be improved

Climate Dynamics

  • ML: FESOM main model in the group, applications generally too large for Albedo, Nice communication with support crew, large problems with the file system

Paleo

  • MWe: Helps to complete project work. Straightforward access to dedicated resource time is important. Usage w/out paperwork review is very positive. Hardware redundancy to the DKRZ HPC system is very good in the event of (partial) failures or problems on one of the two platforms

Atmospheric Physics

  • RK: ICON users. Appreciate flexibility, archiving works good, super stable system. Problems: Filesystem, DKRZ used due to full availability of necessary data there for regional modelling

Phys. oze

  • CW: FESOM-ISSM coupling via MATLAB, setup only works on Albedo and doesn’t run on DKRZ due to lack of MATLAB
  • MCvC: Positive aspects: no need for paperwork, easy to use.  
  • RT: agrees

Sea ice physics

  • FK: Mostly working with observations, focus not on numerical models, rather data processing is important. Albedo is moderately easy to use and access. GitLab is good because it is mounted. Support People are easy to contact.
    • Don't like: more GPU, more personal storage space, people are not using the tape, need some training on using. The VPN connection is often unstable. 

A3m-transform (Young investigators) 

  • DF: No additions, share of usage similar in the future

Computing

  • JK: Good documentation, Easy to access, good overview of GPU usage
  • LN: Waiting times are small, stable system, good support, easy to use, easy to get resources for INSPIRES projects.

Chat comments

MT:

  • With respect to connecting data from Potsdam: 
  1.   Mounting of isipd is NOT a good idea, because that could lead to the impression that the data is fastly available (which it isn't -> Bottleneck of 10 Gb).
  2.   Following the rule that "compute where your data is", the solution is to move (or copy) the data to Bremerhaven (Albedo or Isilon).
  • With respect to "parallel file system is unstable": YES, that is true 🙁  We (HPC) are not satisfied either. But NEC and IBM have not found the origin of this problem, although we insist again and again 🙁 .
  • With respect to "working with tape": 
  1.   SAVING/ARCHIVING is not different than any other remote FS. Just copy your data with rsync, and you are fine. I don't understand what support could be needed here? 
  2.   READING data from the tape needs the understanding that a) that /hs/projects in fact refers to tape and b) that staging is highly recommended before copying the data. If you have any suggestions on how to improve the existing HSM documentation for inexperienced users, I am very happy to consider that 🙂.

Presentation 

Presentation Continued by MAM:

  • Future HPC options - some details and some disclaimers.

Presentation Comments

Option comparison comments:

  • FK: Why not keep Albedo? 
    • SF: There is a 6-year contract limitation, and the support and maintenance are issues afterward.
  • LN: Maintenance contracts are limited and difficult to keep the Albedo. Numbers provided by vendors are very uncertain, mostly list prices.
  • SF: Scaling energy consumption/costs for given performance is difficult as well. 
  • MAM: Currently, 64 Cores/CPU, Turin/Venice will provide about twice the number.
  • LN, SF: Power consumption must be considered in light of available CPUs.

Discussion

Discussion on DKRZ-hosted vs. on-premises:

Q: What advantages/disadvantages/risks do you see in an HPC hosted locally at AWI?

Q: What advantages/disadvantages/risks do you see in an AWI-dedicated HPC hosted at DKRZ?


Marine Bio. Geo.:

  • OG: DKRZ good option compared to Albedo, preference for Albedo to be continued, needed for smaller simulations, which are frequent in the group, and for redundancy of HPC resources
  • CV: Agrees, appreciates reliability, seemingly has more problems with DKRZ, prefers a local solution.

Deep-Sea:

  • CB: In-house support is positive.
  • SF: Software stacks at DKRZ are not naturally available in all fields present at AWI.

Glaciology:

  • TK: Models can run everywhere. However, all applications involving data need a large bandwidth, a strong preference for in-house continuation 
  • VH: Local software support essential, bandwidth 

Perma:

  • JN: An in-house solution is more attractive, and newcomers easier to get used to.
  • IN: Flexibility and ease of use are good advantages for local HPC.

Climate Dynamics:

  • ML: For FESOM, usage of Albedo is not super important; nevertheless, personal preference for an in-house solution. In the FESOM group, there might be a preference for moving FTEs from hardware to model development support

Paleoclimate: 

  • MWe: Agree with ML. If there is no clear gain in computing power, preference for a local solution. FTE computation includes the procurement and documentation of the new system.
  • SF: Procurement can be streamlined, similar to other systems.

Atmos. Phys.: 

  • RK: Both options work; having DKRZ’s Data pool would allow them to use it for regional simulationsimportant.
  • WD: Local system not mandatory. How about increasing the AWI share at DKRZ?
    • SF: Politically complicated. 
    • MAM: FTEs spent on a smaller system are virtually the same, so it is not cost-effective (same work for a smaller computing capacity locally). 
  • On the other hand, the redundancy factor is also mentioned

Phys. Oze:

  • MCvC: Necessary jobs could be run anywhere (however, depending on Matlab). Stability issue needs to be considered, local system as a backup  (Who has 2 has 1, who has 1 has none 😀)
  • RT: Not many things to add. Support for the model and the HPC system combined is of great importance

Sea-Ice Physics:

  • FK: The majority of the section supports a local system.

A3m-transform:

  • DF: Redundancy is an important aspect. 

Data:

  • JK: Work at  DKRZ is possible, however comes with frictions, some preference for an in-house solution, data transfer between DKRZ/AWI/Potsdam. Sustainability aspects to be taken into account as well

Computing:

  • LN: Flexibility of the local system has advantages. With an HPC system, AWI is a player in the national HPC community, otherwise a loss of access to the relevant discussions. 
    • SF: The profile changes from HPC owner to HPC user.
    • RT: If we move to DKRZ now, will it be possible to get back to the local solution?
    • LN: Expertise and competency for running the local system will be gone

Presentation

Wrap-up by MAM:

  • Timeline and closing remarks 
  • No labels