Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Your (larger) Data is archived on a tape, so it will take a while to get it back (unless it is online or has a disk-copy).
  2. You need a project to archive your data (apply for a project here: eResources).

HSM at AWI

Four domains
of
for tape archiving at AWI

Image Modified


DomainUsage
Purpose
A(mainly) PANGAEA and
SENSOR
Registry
Permanent archive for irrecoverable data. Metadata is needed for this domain. Please
contact Stefanie Schumacher or Janine Felden for more information aboutPANGAEA and how
contact PANGAEA on how to submit your data
POpen to all users
Long term project data with predefined life time. A project can be created with eResources https://cloud.awi.de/#/projects 
The setgid on directories ensures that new (sub-)directories belong to the project (POSIX) group automatically
IB, IPDesaster recovery
Daily (or weekly) automatic replication of online project data from the Isilon (when selected in eResources) in Bremerhaven and Potsdam, respectively
DInternal IT stuff
For internal IT use (used for additional backups and storing of expired user/project data)



DomainFile SystemsDisk CopyDescriptionHow to apply

A

/hs/platforms

(tick)

Archive of sensor.awi.de

Only possible with sufficient metadata, see links.
/hs/usero
/hs/pangaea
(tick)PANGAEA
P

/hs/projects
/hs/s3projects

(tick)Project data

Please use eResources to create a project and request storage resources. 

C

/hs/usera
/hs/userc

/hs/userm


Have vanished early 2020


IB, IP/hs/isirep-...(minus)Project replica from the Isilons (Bhv, Pot)Disaster recovery only
D/hs/backup(tick)samfs-dumps, ScoutFS-dumps, logfiles, VeeamIT internal use only
/hs/store(minus)10-year storing of expired user and project data



A disk copyis available for specific (smaller) files for some file systems. This allows a fast access of offline files. The availability of this disk archive depends on the actual resources/usage and might change.

Concept

Principle Idea 

  • HSM:  A (H)ierarchical (S)torage (M)anagement system consists of (at least) two storage systems: A cache speeds up access, and the hierarchy reduces cost. Based on a set of rules, data is stored on certain connected storage devices (tapes and optional disks).
  • ScoutFS is used at AWI since 2024 and supersedes samfs (used since decades before)  www.versity.com/products/scoutfs/  www.scoutfs.org
  • ScoutAM is used as (A)rchive (M)anager www.versity.com/products/scoutam/ and supersedes OHSM/VSM

...