Introduction

The HSM (Hierarchical Storage Management) at AWI provides 

1. (nearly) unlimited storage space and
2.   two replicates on tape (in different buildings for security) and
3. a third copy on disk (for selected data, smaller files for faster access) → VTL (Virtual Tape Library, Silent Brick)
4. for a (comparable) low cost
5. In most cases we can restore data from many years ago (as long as we have the tapes and a compatible drive to read them). If you need an old version, please provide the full path/filename and the date. 

However, there are some caveats:

• Your data is archived on tape and if it is offline, it might take a few minutes to stage it (= to copy it back to the disk cache).
• You need a project to archive your data (apply for a project here: eResources).
•   When you copy a lot of data (>10 TB) in your HSM project directory you might exceed your online quota before the disk cache is released automatically. You will get an automated email, explaining the problem. To solve this this you can release online data explicitly (seek for "release" at HSM: User interface#Specifictasks). Note: This is a save operation, because you can only release data that has already been saved on tape  .

HSM at AWI

Four domains for tape archiving at AWI

Domain Path Disk  Copy Short Description Purpose A /hs/D-A/platforms Archive of o2a https://registry.o2a-data.de/ Permanent archive for irrecoverable data. Metadata is needed for this domain. Please contact  PANGAEA on how to submit your data. /hs/D-A/pangaea /hs/usero PANGAEA Note: Former /hs/usero has moved to /hs/pangaea/data/legacy/ on 2024-06-17 P /hs/D-P/projects /hs/D-P/s3projects Project data Long term project data with predefined life time. A project can be created (or extended) with eResources https://cloud.awi.de/#/projects  The setgid on directories ensures that new (sub-)directories belong to the project (POSIX) group automatically. Please use eResources to create a project and request storage resources.  I /hs/D-I/... Desaster recovery: Project replica from the Isilons (Bhv, Pot) Daily (or weekly) automatic replication of online project data from the Isilon (when selected in eResources) in Bremerhaven and Potsdam, respectively. Used for disaster recovery only (=hopefully never ). R /hs/D-R/backup samfs-dumps, ScoutFS-dumps, logfiles, Veeam IT internal use only /hs/D-R/store 10-year storing of expired user and project data /hs/D-R/veeam VM(Veeam) backups  /hs/s3gateway experimental storage Z /hs/D-Z/global   No archiving Internal parallel filesystem C /hs/usera /hs/userc /hs/userm Have vanished early 2020 A disk copy is available for specific (smaller) files for some file systems. This allows a significantly faster access of offline files. The availability of this disk archive depends on the actual resources/usage and might change.

Concept

Principle Idea 

• HSM:  A (H)ierarchical (S)torage (M)anagement system consists of (at least) two storage systems: A cache speeds up access, and the hierarchy reduces cost. Based on a set of rules, data is stored on certain connected storage devices (tapes and optional disks).
• ScoutFS is used at AWI since 2024 and supersedes samfs (used since 2004)  www.versity.com/products/scoutfs/  www.scoutfs.org . 
• ScoutAM is used as (A)rchive (M)anager www.versity.com/products/scoutam/ and supersedes OHSM/VSM

LTO Tapes

• ... are about 1.1 km long
• ... run with ~10m/sec = 36 km/h
• LTO-9 writes with up to 400 MB/sec

The Circle of Life

Archiving

• When creating a file in ScoutFS (e.g., by rsync, scp, ftp, S3-PUT) the data is stored on a fast SSD cache system.
• Depending on predefined policies (e.g., when the file has not been modified for a specific amount of time) the file is automatically archived on slower (and much cheaper) tapes (and optionally HDD).
• A file just created is online

Releasing

• The metadata (filename, size, ownership, permissions, etc.) of a file remains always in the cache system and is visible/accessible, but 
• when the cache system fills up (e.g, 90% capacity) the data of older files is released.
• If the data of the file is released it is called offline.
• The user does (in the first instance) not see any difference between an online and an offline file.

Staging

• When offline data is accessed, ScoutAM intercepts the call and automatically gathers the data from the archive media. ScoutAM uses information stored in metadata to find (and stage = "copy back") the data.
• In the meantime the read from this file is blocked, thus the process accessing the data is blocked, too.
• When accessing more than a few files, prior staging is strongly recommended (see HSM: File Access User commands)!

Recycling

• If the content of a file changes a new archive copy has to be produced. (You can not modify just the relevant bits on the tape.)
• The previous archive copy becomes useless (aside from having an additional backup of a previous version).
• If a file is deleted, the archive copy becomes useless, too.
• Both processes result in unused (invalid) sections on a tape. 
• Eventually only a small part of a tape contains relevant (up to date) information. The residual data can be archived on other tapes, the old tape is erased and can be used for future archive copies. This process is called recycling.

Hardware

Tape Library

• 2 TFinity Tape Libraries in two buildings
• each has 3100 licensed slots for LTO tapes
• 600 LTO-9 tapes in each library (600x ~20 TB → ~12 PB)

TFin-E	TFin-D
Live View → Frame #2 or Frame #3 User: viewer PW: Viewer123!	Live View → Frame #2 or Frame #3 User: viewer PW: Viewer123!

Server, Switches and disks

• 3x Dell R760xd server (hssrv2[a-c].dmawi.de) with 256 GB RAM
• 2x Dell R760xd server (hssrv3[a-b].dmawi.de) with 512 GB RAM
• 2 Brocade (64Gb) 7710 SAN switches 
• 4 Brocade (32Gb) 6610 SAN switches (two in each building)
• 300 TB NVMe primary cache (Quantum F-2200) 
• 488 TB extended HDD-cache (Dell ME5084-HDD)
• 1120 TB Virtual Tape Library (VTL) Silent Brick von Fast-LTA 
  • Controller G5200
  • SAS Switch
  • 7x SilentBrick Max