Every 4th Thursday in a month, 14:00-15:00, hybrid (room H-1049) or online

https://awi.webex.com/awi-en/j.php?MTID=m66b8d3e371a2d48a5356fe3c50751cf7

To Remind users:

Email to: VL_HPC_user@awi.de

Previous meeting's slides and materials: https://nextcloud.awi.de/s/nBMm2yMKeS3rYyS


We will always start with a short informative section. If you would like to present your cool HPC-project, this is also an option!

The basic idea is to have an informal round for comments and questions on HPC in general and at AWI in particular.


DateTopicSlides / Comments / Material

22nd Aug 2024

  • Announcements
    • Conda licensing updates
      • do not use defaults channel in conda
      • instead, use conda-forge
      • conda-forge is used by default in conda's modules in Albedo
    • Disruption in albedo0 (login node) - due to maintenance
      • for the time being use albedo1
  • Q&A
    • What should I do if I have existing conda environments to avoid using the defaults channel?
      • run conda list and check if you have anything installed from the defaults channel. If you do you can:
        • conda config --remove channels defaults
          conda config --add channels conda-forge
          conda config --set channel_priority strict
          
           
        • If you are still seeing the defaults channel (or no channel at all) after your conda list command, you'll have to rebuild your environment. If you have any problems by doing so write us an email to hpc@awi.de

hybrid

H-1049

27th Jun 2024

hybrid

H-1049

Slides: https://notes.desy.de/p/Q2I-7FXT0#/

Slides as documentation: https://notes.desy.de/tPTGtbBMRB-tWOHcCtiq3Q?view

Recording: https://nextcloud.awi.de/s/bxxadbF9AwsYssH

23rd May 2024

hybrid

H-0049

Slides: https://nextcloud.awi.de/s/n2mng3F9Ey3Pcff

25th Apr 2024

  • S3 object storage in HSM: prototype (Malte Thoma and Pavan Kumar Siligam )
  • Q&A
  • Presentation: AI/ML and climate analysis - it all starts with data Christopher Kadow, DKRZ
    • Christopher is the leader of the Data Analysis group at DKRZ. Among other tasks his group have developed the NextGEMS data portal, develop analysis tools, they are organizing Machine Learning workshops, working on the future DKRZ object storage, workflows with zarr data format...
    • Abstract: 

      This talk gives an overview of the topics of the new department Data Analysis at DKRZ - using new technologies for climate research and data handling on the HPC Levante. We demonstrate the successful transition of an AI image processing method into a climate data reconstruction application. In addition to exciting results of reconstructed climate events in the distant past dating back to 1850, the importance of the data base in our field will be addressed. Understanding the content of the data collection is key to the proper application of the latest AI/ML and climate analysis. Linking different data sources for such target variables is part of the BMBF projects ClimXtreme, RegiKlim and WarmWorld - e.g. www.gems.dkrz.de . The results presented link observations with climate model simulations and, in combination with ML, provide an unprecedented insight into different climate applications such as data assimilation, downscaling and digital twins.
      Kadow, C., Hall, D.M. & Ulbrich, U. Artificial intelligence reconstructs missing climate information. Nat. Geosci. 13, 408–413 (2020). https://doi.org/10.1038/s41561-020-0582-5

    • You can request a consultation session with him on that day!
hybrid

H-0049

Slides: https://nextcloud.awi.de/s/JHw5HXaxmRK53cM

28th Mar 2024 → 11th Apr 2024

hybrid

22nd Feb 2024

  • news about JupyterHub
  • File system news
  • Coming meetings and presentations
  • Q&A
  • Presentation: HPC and Data Process Service Catalogue - Miguel Andrés-Martínez
hybrid

Slides:
https://nextcloud.awi.de/s/wPDd9KrrPsfFd9p

25th Jan 2024

  • Advertisement of pool-tool workshop
  • GPFS (file system) issues and work in progress
  • New analysis-toolbox/01.2024
  • Jupyter on Albedo
Online

23rd November 2023

  • Albedo support service update
  • New Analysis-Toolbox in December
  • JupyterHub work
    • testing phase
  • Demo by Sonal Rami: Machine learning overview: comparing CPU, GPU and Multi-GPU performance
hybrid

Link to the recorded presentation: https://nextcloud.awi.de/s/oxmjEWg8AzHWHDR

Slides and Python script for the demo:
https://github.com/SonalRami/ML_demo-CPU-vs-GPU

26th October 2023

  • Current support shortage disappointed face 
  • Last GPUs arrived
  • User interviews and Fall plans for the group
  • Updates on
    • Spack 0.18.1 → 0.20.1 (Done!)
    • gprofng (Done!)
    • IDL-8.9.0/ENVI-5.7 (Done!)
  • Documentation (Software: R, JupyterLab/Hub..., WIP)
    • conda environments/python environments, spack documentation, guidelines (WIP)
  • JupyterHub
    • Invite users that use JupyterLab to request joinning the testing phase
    • R in JupyterHub
    • Consoles not working! (error) 
  • From the previous meeting
    • Albedo Queue information (WIP)
      • statistics? (not urgent)
        • idle time/processing time...
        • Malte → sacct, sreport
    • Directories permissions
      • by default be able to read work directory?
        • Malte → all employees would be able to read, not safe. Use eresources projects
    • FESOM-1 status report Albedo
    • CMORization status
    • Pools for modelling updates
      • Miguel → not our responsibility
      • Jan → provide tools to mirror pools
      • Malte → which one is the leading pool?
  • Next Meeting in November
    • ML demo with TensorFlow-Keras and GPUs, by Sonal Rami
 online

28th September 2023

  • Current support shortage disappointed face 
  • Updates on
  • JupyterHub
    • Invite users that use JupyterLab to request joinning the testing phase
    • R in JupyterHub
online

25th August 2023

  • Albedo updates
  • JupyterHub updates
  • New team member
hybrid, you can also join us in person: room H-1049

27th July 2023

Meeting cancelled due to holidays of multiple team members and another simultaneous meeting


22nd June 2023

  • Introduction round
  • HPC Users Interviews announcement
  • Albedo news
    • Installation of new GPU node
  • Ollie shut down
  • JupyterHub progress
hybrid, you can also join us in person: room H-1049

25th May

Monday, May 22nd 2023

  • upcoming changes in the HPC team
  • EOL Ollie approaching → last chance to save your data!
  • QnA
hybrid, you can also join us in person: room H-1049
27th April 2023???
23 March 2023only online
23 February 2023
  • Albedo:
    • Future of hyperthreading nodes on Albedo (smpht, mppht)
    • Extension of GPU partition: 2 more nodes (Preferences on hardware?)
    • Slurm Powersafe
    • Parallel I/O
    • Need for QOS "xlarge" (ollie) → 1wk (albedo)?
  • QnA
Hybrid meeting: Room Arktis G-301 & Webex
26 January 2023
no meeting
22 December 2022QnA
24 NovemberCurrent status of Albedo
27 Octoberno meeting - holidays again (yes, in between, we do work!)Albedo: slow progress, but impossible to give an exact date for power user access.
22 September 2022no meeting - vacations  & no updates on Albedo
1 September 2022
(postponed by 1 week)
  • Albedo timeline: delays... no user access before October
  • Support team: Natalja will leave AWI end of October → reduced support capacity → strong focus on Albedo basics.

28 July

summer holidays - no meeting


23 June 2022

News on albedo...

  • Hardware has arrived at AWI server room
  • The floor has to be reinforced: 4 July
  • 5 July - ... installation, finally!
  • Plans for 5PB file system: /albedo/work and /albedo/scratch, both with personal and project directories.
    Space for projects via eResources (like /isibhv/projects/)

5 May 2022
(postponed by 1 week)
News on albedo: Delivery from 17th to 25th Mai (smile)
Ollie: Accounting & Job Priorities

Slurm on Ollie → Scheduling
24March 2022Ollie: Bright+CentOS 7.5 → xCat + CentOS 7.9Partitions mpp_new, smp_new
10 March 2022
(as an exception)

27 January 2022AlbedoAlbedo_2022-01-27.pdf