HPC meetings

• hdf5/1.14.3-openmpi4.1.6-intel2021.6.0-4wo2tuf

• netcdf-c/4.8.1-openmpi4.1.6-intel2021.6.0-wucmueg

• netcdf-fortran/4.6.1-openmpi4.1.6-intel2021.6.0-irbf4k2

• netcdf-cxx/4.2-openmpi4.1.6-intel2021.6.0-yyjdfih

• eccodes/2.34.0-openmpi4.1.6-intel2021.6.0-wm2pijy

• openmpi/4.1.6-intel2021.6.0

• hdf5/1.14.3-openmpi4.1.6-intel2021.6.0

• netcdf-c/4.8.1-openmpi4.1.6-intel2021.6.0

• netcdf-fortran/4.6.1-openmpi4.1.6-intel2021.6.0

• netcdf-cxx/4.2-openmpi4.1.6-intel2021.6.0

• eccodes/2.34.0-openmpi4.1.6-intel2021.6.0

• fixes problems with Cartopy:
  ERROR 1: PROJ: proj_create_from_database: Open of /albedo/soft/sw/conda-sw/analysis-toolbox-01.2026/share/proj failed."

• isilon connection from Albedo → Solved
• intermitent pmix/ucx problems with applications with large numbers of tasks → under investigation

  • 0: slurmstepd: error: mpi/pmix_v3: pmixp_p2p_send: prod-164 [0]: pmixp_utils.c:467: send failed, rc=2, exceeded the retry limit 300: slurmstepd: error: mpi/pmix_v3: _slurm_send: prod-164 [0]: pmixp_server.c:1583: Cannot send message to /var/spool/slurmd/stepd.slurm.pmix.42881068.0, size = 50253, hostlist: 31248: slurmstepd: error: mpi/pmix_v3: pmixp_coll_ring_reset_if_to: prod-169 [2]: pmixp_coll_ring.c:742: 0x15553c012d10: collective timeout seq=0 33248: slurmstepd: error: mpi/pmix_v3: pmixp_coll_log: prod-169 [2]: pmixp_coll.c:281: Dumping collective state 34248: slurmstepd: error: mpi/pmix_v3: pmixp_coll_ring_log: prod-169 [2]: pmixp_coll_ring.c:760: 0x15553c012d10: COLL_FENCE_RING state seq=0 35
    

• intermitent slow-down on reading files with netcdf module(s) → under investigation
  
  • Thomas Kleiner → problems on the past when compiling an application with Intel-MPI → it stalled while reading netcdf files. Using OpenMPI works (module load openmpi/4.1.3)

Wrap-up of the system update to Rocky Linux 8.10

ushow available as a module on albedo

• More details on the software: https://github.com/CliDyn/ushow

New analysis-toolbox available (01.2026) but with problems with cartopy (workarounds and update)

Helmholtz Software Quality Indicator: information session on May 8th, 10-12, Register here via Indico

• Remark on this topic: If you are using esm-tools for your runs, you may consider citing us: https://github.com/esm-tools/esm_tools/blob/release/CITATION.cff

• Maintenance on the day by NEC
• Status update

• We will run it again soon (~March)
• RZ Workshops: https://events.hifis.net/category/218/

• Jacamar contraints: external users cannot trigger the CI because they don't have HPC accounts
  • Paul Gierz is looking into that. We would probably never be able to allow external users to trigger runners on the HPC because of legal restrictions

• lustre_du 
• new VAST file system
• new HSM system to be deployed soon
• for compression a recommendation: use zstd is faster and better

• Feedback:
  • Documentation on GPU monitoring in JupyterHub is missing. Add to the documentation that one can do it directly from the terminal in JupyterHub or with the ! to run shell commands from the notebook (Jonas Küpper )
  • Mirjam Bourgett has a script to installing fonts that will share with the HPC people so that we can make it available in the documentation

• Raphael Köhler used to have problems when running with 20 nodes but now he is running with 10 and he is experiencing no issues
• Mirjam Bourgett and Hugo Lévy reported problems when:
  • changing branches in git
  • opening text files some times is slow
  • MIT-GCM can be slow even when running with 1 node for tests
  • it is better than it was one month ago or so

• rsync, there are other tools like rclone but Miguel Andrés-Martínez  have not personally tried it

• https://spaces.awi.de/spaces/HELP/pages/426525152/System+overview+-+albedo+s+hardware#Systemoverviewalbedo'shardware-Computenodes

• in Albedo https://cloud.awi.de/#/projects > your project > Storage Resources > file permissions icon
• in other HPCs in shared folders
  • for directories: find /your/path -type d -exec chmod 2750 {} +
  • for files: find /your/path -type f -exec chmod 640 {} +

• High utilization of CPU part except weekends and holidays
• FREVA - https://docs.dkrz.de/doc/dataservices/finding_and_accessing_data/freva/index.html
  
  • freva.dkrz.de
  • browse data at DKRZ
  • interesting plugins and ChatGPT interface for data search and developing specific analysis scripts and 
• Tender procedure HSM →  preparational meetings with the vendors finished, in June request of proposals
  
  • will be procured at the end of the year
• HLRE 5 → with the budget they can only afford the same CPU capacity
  
  • they are planing to keep 1/3 of Levante running in parallel
  • new CPU nodes + old levante = 80% of current levante CPU capacity 
  • new GPU nodes as much as available for the remaining money
  • storage 60 PB
  • Benchmarks ready to be published (CPU: FESOM, ICON and EMAC, GPU: ICON + some generic benchmarks)

• /data/pool
  
  • users need to acknowledge the use of DKRZ resources in their publications, that includes also the use of the data in the /data/pool 
  • is already included in the Terms of use: "Users must quote and acknowledge the use of DKRZ resources in all publications related to their production and development projects as follows:

    This work used resources of the Deutsches Klimarechenzentrum (DKRZ) granted by its Scientific Steering Committee (WLA) under projectID ###

    Add the following line if data from /pool/data have been used:

    Further, datasets provided by XXX via the DKRZ data pool were used."

  • cite the used dataset, bib-files are provided by the owners of the datasets

• please, do not "rejuvenate" files on scratch automatically
• ask for help if you need to discuss/advice for your storage strategy → hpc@awi.de

• we have support bought until mid 2027, but we will almost certainly extend the support at least 1 year but possibly 2 years. That puts us in n earlier estimate of 2028 and a later estimate of 2029 for it's decommission.

• do not use defaults channel in conda
• instead, use conda-forge
• conda-forge is used by default in conda's modules in Albedo

• for the time being use albedo1

• run conda list and check if you have anything installed from the defaults channel. If you do you can:

  • conda config --remove channels defaults
    conda config --add channels conda-forge
    conda config --set channel_priority strict
    

     
  • If you are still seeing the defaults channel (or no channel at all) after your conda list command, you'll have to rebuild your environment. If you have any problems by doing so write us an email to hpc@awi.de

• Not complete as in Levante, but sufficient for Alexander Robinson projects
• ERA5 data structure from the cli tool

This talk gives an overview of the topics of the new department Data Analysis at DKRZ - using new technologies for climate research and data handling on the HPC Levante. We demonstrate the successful transition of an AI image processing method into a climate data reconstruction application. In addition to exciting results of reconstructed climate events in the distant past dating back to 1850, the importance of the data base in our field will be addressed. Understanding the content of the data collection is key to the proper application of the latest AI/ML and climate analysis. Linking different data sources for such target variables is part of the BMBF projects ClimXtreme, RegiKlim and WarmWorld - e.g. www.gems.dkrz.de . The results presented link observations with climate model simulations and, in combination with ML, provide an unprecedented insight into different climate applications such as data assimilation, downscaling and digital twins.
Kadow, C., Hall, D.M. & Ulbrich, U. Artificial intelligence reconstructs missing climate information. Nat. Geosci. 13, 408–413 (2020). https://doi.org/10.1038/s41561-020-0582-5

• Write an email to Bernadette Fritzsch and Miguel Andrés-Martínez for requesting that
• He'll be available between ~10:00 until the talk (~14:00) for consultation sessions

• statistics? (not urgent)
  • idle time/processing time...
  • Malte → sacct, sreport

• by default be able to read work directory?
  • Malte → all employees would be able to read, not safe. Use eresources projects

• Miguel → not our responsibility
• Jan → provide tools to mirror pools
• Malte → which one is the leading pool?