...
The HSM (Hierarchical Storage Management) at AWI provides
- (nearly) unlimited storage space and
- two replicates on tape (in different buildings for security) and
- a third copy on disk (for selected data, smaller files for faster access) → VTL (Virtual Tape Library, Silent Brick)
- for a (comparable) low cost
However, there are two caveats:
- Your (larger) Data is archived on tape, so it will take a few minutes to get it back (unless it is online or has a disk-copy).
- You need a project to archive your data (apply for a project here: eResources).
HSM at AWI
Four domains for tape archiving at AWI |
---|
Image Modified | |
Usage | | Purpose | Path | Disk Copy | Description | How to apply |
---|
|
Purpose |
(mainly) PANGAEA and RegistryPANGAEA | /hs/platforms |
| Archive of sensor.awi.de | Permanent archive for irrecoverable data. Metadata is needed for this domain. Please contact PANGAEA |
|
on how on how to submit your data. | /hs/usero /hs/pangaea | | PANGAEA Note: Former /hs/usero has moved to /hs/pangaea/data |
|
P | /legacy/ on 2024-06-17 | P | Projects | /hs/D-P/projects /hs/D-P/s3projects | | Project data |
|
Open to all users |
IB, IP | Desaster recovery | Daily (or weekly) automatic replication of online project data from the Isilon (when selected in eResources) in Bremerhaven and Potsdam, respectively | D | Internal IT stuff | For internal IT use (used for additional backups and storing of expired user/project data) | Domain | Path | Disk Copy | Description | How to apply |
---|
A | /hs/platforms |
| Archive of sensor.awi.de | Only possible with sufficient metadata, see links |
/hs/usero /hs/pangaea | | PANGAEA |
P | /hs/projects /hs/s3projects | |
Project data | Please use eResources to create a project and request storage resources. | C | Depricated
| /hs/usera /hs/userc /hs/userm |
| Have vanished early 2020 |
|
IB, IP |
isirep-...D-I/ | | Project replica from the Isilons (Bhv, Pot) |
|
Disaster recovery only | D |
fast significantly faster access of offline files. The availability of this disk archive depends on the actual resources/usage and might change. |
Concept
Principle Idea
- HSM: A (H)ierarchical (S)torage (M)anagement system consists of (at least) two storage systems: A cache speeds up access, and the hierarchy reduces cost. Based on a set of rules, data is stored on certain connected storage devices (tapes and optional disks).
- ScoutFS is used at AWI since 2024 and supersedes samfs (used since 2004) www.versity.com/products/scoutfs/ www.scoutfs.org .
- ScoutAM is used as (A)rchive (M)anager www.versity.com/products/scoutam/ and supersedes OHSM/VSM
...
- 2 TFinity Tape Libraries in two buildings
- each has 3100 licensed slots for LTO tapes
- 600 LTO-9 tapes in each library (600x ~20 TB → ~12 PB)
Server, Switches and disks
- Three Dell R760xd server (hssrv2[a-c].dmawi.de)
- 4 Brocade (32Gb) 6610 SAN switches
- 270 TB SSD primary cache (Dell ME5084-SSD)
- 488 TB extended HDD-cache (Dell ME5084-HDD)
- Ab Ende Planed for end of 2024:
1120 TB Virtual Tape Library (VTL) Silent Brick von Fast-LTA - Controller G5200
- SAS Switch
- 7x SilentBrick Max
...