...

• Your (larger) Data is archived on tape, so it will take a few minutes to get it back (unless it is online or has a disk-copy).
• You need a project to archive your data (apply for a project here: eResources).

  During the transition (in 2024) from hssrv1 (samfs) to hssrv2 (ScoutAM) resources are limits because:

  Existing Tape drives (six per building) are split between old and new

HSM

...

We have to copy (rearchive) 600 tapes due to a tape recall

...

The new ScoutAM does not support automatic unloading after a certain idle time yet. Hence, all drives are occupied and before a stage request can start one drive has to be unloaded (and the tape inside must be rewound first).  This might take a few minutes.

...

HSM at AWI

IB, IPisirep-...

Four domains for tape archiving at AWI
Image Modified	Domain Purpose Path Disk Copy Description How to apply A PANGAEA /hs/platforms Archive of sensor.awi.de Permanent archive for irrecoverable data. Metadata is needed for this domain. Please contact PANGAEA on how to submit your data. /hs/usero /hs/pangaea PANGAEA Note: Former /hs/usero has moved to /hs/pangaea/data/legacy/ on 2024-06-17 P Projects /hs/D-P/projects /hs/D-P/s3projects Project data Long term project data with predefined life time. A project can be created with eResources https://cloud.awi.de/#/projects  The setgid on directories ensures that new (sub-)directories belong to the project (POSIX) group automatically. Please use eResources to create a project and request storage resources.  C Depricated /hs/usera /hs/userc /hs/userm Have vanished early 2020
I Desaster recovery /hs/
D-I/ Project replica from the Isilons (Bhv, Pot) Daily (or weekly) automatic replication of online project data from the Isilon (when selected in eResources) in Bremerhaven and Potsdam, respectively. Used for disaster recovery only (=hopefully never ). D Internal IT stuff /hs/backup samfs-dumps, ScoutFS-dumps, logfiles, Veeam IT internal use only /hs/store 10-year storing of expired user and project data /hs/s3gateway experimental storage A disk copyis available for specific (smaller) files for some file systems. This allows a significantly faster access of offline files. The availability of this disk archive depends on the actual resources/usage and might change.

Concept

Principle Idea 

• HSM:  A (H)ierarchical (S)torage (M)anagement system consists of (at least) two storage systems: A cache speeds up access, and the hierarchy reduces cost. Based on a set of rules, data is stored on certain connected storage devices (tapes and optional disks).
• ScoutFS is used at AWI since 2024 and supersedes samfs (used since 2004)  www.versity.com/products/scoutfs/  www.scoutfs.org . 
• ScoutAM is used as (A)rchive (M)anager www.versity.com/products/scoutam/ and supersedes OHSM/VSM

...

• 2 TFinity Tape Libraries in two buildings
• each has 3100 licensed slots for LTO tapes
• 600 LTO-9 tapes in each library (600x ~20 TB → ~12 PB)

Image Modified TFin-E	Image ModifiedTFin-D
Live View → Frame #2 or Frame #3 User: viewer PW: Viewer123!	Live View → Frame #2 or Frame #3 User: viewer PW: Viewer123!

Server, Switches and disks

...