HSM at AWI

TFin-E

The HSM (Hierarchical Storage Management) provides for your data

  1. unlimited storage space and
  2. two replicates on tape (in different buildings)
  3. a third copy on disk (for selected data, smaller files)

However, there are two caveats:

  1. Your Data is archived on a tape, so it will take a while to get it back (unless it is online or has a disk-copy).
  2. You need a project to archive your data (apply for a project here: eResources).


TFin-D

← Live View
User: viewer PW: Viewer123!

Suggestions/good to know

  1. Storing millions of files in one directory slows down any filesystem, in particular SamFS. Therefore we suggest to limit the number of files whenever possible (per design, workflow, zip, tar, ...)
  2. We suggest to limit the size of each individual file to a maximum of 500 GB per file. E.g. with
    tar -cPf - $DIRECTORY | pigz -c | split -a3 -d -b500GB - $TOFILE.
  3. If you access(read) more than one file from the HSM, please stage all files you plan to access at once! This will decrease the overall access time significantly by reducing the necessary robotic accesses to a minimum.
Four domains of tape archiving at AWI


DomainUsagePurpose
A(mainly) PANGAEA and SENSORPermanent archive for irrecoverable data. Metadata is needed for this domain. Please contact Stefanie Schumacher or Janine Felden for more information about PANGAEA and how to submit your data
POpen to all usersLong term project data with predefined life time. A project can be created with eResources https://cloud.awi.de/#/projects 
The setgid on directories ensures that new (sub-)directories belong to the project (POSIX) group automatically
IB, IPDesaster recoveryAutomatic replication of online project data from the Isilon (when selected in eResources) in Bremerhaven and Potsdam, respectively
Dinternal IT stuffFor internal IT use (used for additional backups and storing of expired user/project data)
DomainFile SystemsDisk Copy
How to apply

A

/hs/platforms


(tick)


Archive of sensor.awi.de

Only possible with sufficient metadata.

/hs/usera


(tick)


Archive of individual Projects

/hs/usero
/hs/pangaea
(tick)PANGAEA
P

/hs/projects

(tick)Project data

Please use eResources to create a project.

C

/hs/userc
/hs/userm

No

Have vanished since early 2020


IB, IP/hs/isirep-...(minus)Project replica from the Isilons (Bhv, Pot)Disaster recovery only
D/hs/backup(tick)samfsdumps and logfilesIT internal use only
/hs/store(minus)10-year storing of expired user and project data



A disk archive is available for specific (smaller) files for some file systems. This allows a fast access of offline files. The availability of this disk archive depends on the actual resources/usage and might change.

Access to hsm.dmawi.de

Read Access

Files might be offline for several reasons. If you want to access these they are read from tape automatically, but this takes some time. If you want to read/copy more than one file staging is strongly recommended (background see below, command see right column).

The Windows Way

Connect to \\hsm.dmawi.de\ within windows explorer (use the right mouse button and add a network device). You can either search for shared directories or directly connect with e.g., \\hsm.dmawi.de\projects\<project>.

The Linux Way

All HSM file systems are shared and mounted automatically on most Linux clients. A simple ls /hs/projects/ should do. If you miss any directory please contact hsm-support@awi.de

Read/Write Access

The HSM system ist shared/mounted read only. To write data into the tape archive use one of the following options:

SuggestionCommandImportant Notes
(big grin) Best choice

rsync -e ssh -Pauv <file|dir> <username>@hsm.dmawi.de:<destination-dir>

rsync is the most versatile way of transfering data. E.g., it allows updates with the -u option. This ensures that only new files are copied (and overwritten), existing (unchanged) files are not touched. This is important to reduce tape access. You do not want to use -a, because this would  stage all files from tape to the disk-cache for a complete file-comparison.

When copying directories you need -r (recursive, already included in -a).

(big grin) Fast choicesftp
filezilla
sftp provides fast way of transferring large amounts of data. Use your favourite ftp-client. However, note, that only two connections per user are allowed. If you request more, your connection will terminate. sftp uses the secure ssh-protocol and should be preferred. Use port 22 for sftp.
(minus) Do not use!scp <file|dir> <username>@hsm.dmawi.de:<destination-dir>scp seems convenient, but it is slightly slower when transferring data compared to ftp and/or rsync. It also just overrides existing files and no update (like rsync -u) is possible. This would also create new tape copies, you do not want to do that!!!

Note: If you have to archive many (>100 000) small (<100 MB) files this will stress the system more than necessary. Please zip or tar[.gz] your directories and upload these compressed files.

Execute commands on hsm.dmawi.de

Direct access (login) to hsm.dmawi.de is not possible. However, you can execute remote commands on hsm.dmawi.de in a restricted shell to get information about your data. E.g., you can release and stage your data if necessary. See the right column for some useful commands. They are executed with: ssh <username>@hsm.dmawi.de <command>

Change the permissions

Linux way:
As file systems are mounted read only, you have to execute a chmod on the server. As permissions for directories and files are slightly different (most files should not have the 'x') the following suggestion might be useful to restore the default permissions:

ssh hsm.dmawi.de "find /hs/<DIR> -type d -exec chmod 2775 {} \; "  # directories get  drwxrws---
ssh hsm.dmawi.de "find /hs/<DIR> -type f -exec chmod 0664 {} \;  " # files get -rw-rw----

Windows way
File permissions can be changed with filezilla or another sFTP program.

Create an ssh-key for hsm.dmawi.de

Your $HOME on hsm.dmawi.de is the standard UNIX home directory. You can use a ssh-key for hsm.dmawi.de. Execute these commands in a terminal (e.g., putty on windows):

  • Execute >ssh-keygen -t rsa< on your computer/client and just press enter three times (confirmation of key location and empty passphrase).
  • You can not use ssh-copy-id, because login on hsm.dmawi.de is not possible, hence you need either one of the following approaches:
    1. If you have access to another Linux Server, e.g. linsrv1.awi.de  ($HOME is identical): ssh-copy-id -i ~/.ssh/id_rsa.pub linsrv1.awi.de
    2. If you want to add your key:
      • scp hsm.dmawi.de:.ssh/authorized_keys /tmp/$$
      • cat ~/.ssh/*pub >> /tmp/$$
      • scp /tmp/$$ hsm.dmawi.de:.ssh/authorized_keys
    3. If this is your first ssh key:
      • ssh hsm.dmawi.de mkdir .ssh
      • scp ~/.ssh/id_rsa.pub hsm.dmawi.de:~/.ssh/authorized_keys
      • ssh hsm.dmawi.de "chmod 700 ~/.ssh/authorized_keys"

Note: Starting with the 7.0 release of OpenSSH, support for ssh-dsa keys has been disabled by default. You can re-enable support locally by updating your sshd_config (in /etc/ssh, /opt/local/etc/ssh, or ~/.ssh/config) with:

PubkeyAcceptedKeyTypes=+ssh-dss

Some general Information about SamFS & HSM

Principle Idea

  • SamFS stands for (S)torage (a)rchive (m)anager (F)ile (S)ystem
  • SamFS: is a (H)ierarchical (S)torage (M)anagement system (HSM). The HSM consists of two storage systems: The cache to speed up access, and the hierarchy. Based on a set of rules, data is stored on certain connected storage devices (tapes and maybe disks).

The Circle of Life

Archiving

  • When creating a file in SamFS (e.g., by rsync, scp, ftp) the data is stored on a fast cache system (a hard drive).
  • Depending on predefined policies (e.g., when the file has not been modified for a specific amount of time) the file is automatically archived on slower (and much cheaper) tapes.
  • A file just created is online

Releasing

  • The metadata (filename, size, ownership, permissions, etc.) of a file stay always on the cache system and are visible, but 
  • when the cache system fills upl (e.g, 90% capacity) the data of large files and files that have not been touched fro some time is released.
  • If the data of the file is released it is called offline.
  • The user does (in the first instance) not see any difference between a online and offline file.

Staging

  • When offline data is accessed the SamFS intercepts the call and automatically gathers the data from the archive media. SamFS uses informations from the metadata to find the media.
  • In the meantime the reads from this file will be blocked, thus the process accessing the data blocks, too.
  • When accessing more than a few files, prior staging is strongly recommended (see User commands)!

Recycling

  • If the content of a file changes a new archive copy has to be produced. (You can not modify just the relevant bits on the tape.)
  • The previous archive copy becomes useless (aside from having an additional backup of a previous version).
  • If a file is deleted, the archive copy becomes useless, too.
  • Both processes result in unused (invalid) sections on a tape. 
  • Eventually only a small part of a tape contains relevant (up to date) information. The residual data is archived on other tapes, the old tape is erased and can be used for future archive copies.
  • This happens by the following tasks:
    1. The recycler marks the tape and/or files with R.
    2. The next archiver run finds these files and starts re-archiving, the R flag of the file vanishes and a new vsn (volume serial name) for this copy is set.
    3. The recycler recognises that all files are copied somewhere else, because they have a new unique vsn. The old tape gets the status c (old candiadat) and depending on the settings in /etc/opt/SUNWsamfs/recycler.delay an atq job is scheduled for /etc/opt/SUNWsamfs/scripts/recycler.sh.


Related articles

Contact

If you have problems please contact: hsm-support@awi.de

User commands

 To be executed as
ssh hsm.dmawi.de <command>
A detailed explanation to samfs specific commands can be found here: https://docs.oracle.com/cd/E22586_01/html/E22976/glads.html#scrolltoc

User commandDescription
mkdir

Create a new directory, e.g., ssh hsm.dmawi.de mkdir /hs/projects/<project>/newdir

stage

You can (and should) stage a file before you access it. If you use stage -w <file>, the command ends when the file is online. If you want to access more than one file (e.g., a complete directory) you should use stage -r <dir> (recursive) and additionally (optional) stage -r -w <dir> if you want the terminal to wait until all files are staged. NOTE: Never use '-w' in your first stage-command, because you would prevent samfs from optimizing the tape access.
Note: You should never ever stage more data than the cache size allows. Please limit yourself below 20 TB.

https://docs.oracle.com/cd/E22586_01/html/E22976/glajx.html#scrolltoc

sls -D (or sls -2)equivalent to ls, shows detailed information about a file and its archive status, e.g., online or offline: https://docs.oracle.com/cd/E22586_01/html/E22976/glaic.html
sls -E
like sls -D, but shows the md5 checksum, too. Can be used to validate that the file was correctly archived. 
sdu -h

equivalent to du, shows detailed information about a file and its archive status.

https://docs.oracle.com/cd/E22586_01/html/E22976/glaii.html

release

release disk space of archived files if online quota reaches the limit. Try release -r <dir> to recursively release files in (sub-)directories.

https://docs.oracle.com/cd/E22586_01/html/E22976/glaip.html#scrolltoc

sfind

equivalent to find, shows correct information about the file size on tape (and not only on the disk cache).

https://docs.oracle.com/cd/E22586_01/html/E22976/glaia.html#scrolltoc

e.g., is my file online?
ssh hsm.dmawi.de sfind /hs/projetcs/p_xyz -online

saminfo.sh -qGet quota of all groups on HSM
saminfo.sh -sShow staging status
saminfo.sh -tShow tape drive status

A useful combination if you want to get all netcdf-files in a specific directory online would be something like:

ssh hsm.dmawi.de "sfind <absolut-path> -offline -name *.nc -exec stage {} \;"



3 Comments

  1. Malte das Layout der Seite ist nicht grade sehr Benutzerfreundlich  gestaltet. 

    Vor allem der Bereich ganz rechts: User commands ist  IMHO völlig .... weary cat 

    Die  Fotos helfen auch nicht wirklich bei etwas weiter.

    Grüße Wolfgang

    1. Hallo Wolfgang Cohrs, konstruktive Kritik ist immer willkommen slightly smiling face . Historisch gewachsene Dokumente müssen immer mal wieder überarbeitet werden. Leider fehlt mir da aktuell die Zeit für. Hast Du (neben deiner Kritik) auch Vorschläge, wie es besser sein könnte? Die Spaces-Crew kann dir sicher eine Seite mit Schreibrechten einrichten auf der du Vorschläge machen könntest, welches Design dir helfen würde schnell die wichtigen Infos zu bekomemn. 

      Liebe Grüße,
      Malte

  2. Malte (wink)  gute Idee. Aber das ist nicht meine Baustelle.

    Ich würde aber das zweispaltige Layout einfach in einspalten anpassen. 

    m2c