HSM at AWI

TFin-E	The HSM (Hierarchical Storage Management) provides for your data unlimited storage space and two replicates on tape (in different buildings) a third copy on disk (for selected data, smaller files) However, there are two caveats: Your Data is archived on a tape, so it will take a while to get it back (unless it is online or has a disk-copy). You need a project to archive your data (apply for a project here: eResources).	TFin-D
Frame #2 or Frame #3	← Live View → User: viewer PW: Viewer123!	Frame #2 or Frame #3

Suggestions/good to know

1. Storing millions of files in one directory slows down any filesystem, in particular SamFS. Therefore we suggest to limit the number of files whenever possible (per design, workflow, zip, tar, ...)
2. We suggest to limit the size of each individual file to a maximum of 500 GB per file. E.g. with

   tar -cPf - $DIRECTORY | pigz -c | split -a3 -d -b500GB - $TOFILE.

3. If you access(read) more than one file from the HSM, please stage all files you plan to access at once! This will decrease the overall access time significantly by reducing the necessary robotic accesses to a minimum.

Domain	File Systems	Disk Copy		How to apply
A	/hs/platforms		Archive of sensor.awi.de	Only possible with sufficient metadata.
	/hs/usera		Archive of individual Projects
	/hs/usero /hs/pangaea		PANGAEA
P	/hs/projects		Project data	Please use eResources to create a project.
C	/hs/userc /hs/userm	No	Have vanished since early 2020
IB, IP	/hs/isirep-...		Project replica from the Isilons (Bhv, Pot)	Disaster recovery only
D	/hs/backup		samfsdumps and logfiles	IT internal use only
	/hs/store		10-year storing of expired user and project data

A disk archive is available for specific (smaller) files for some file systems. This allows a fast access of offline files. The availability of this disk archive depends on the actual resources/usage and might change.

Access to hsm.dmawi.de

Read Access

Files might be offline for several reasons. If you want to access these they are read from tape automatically, but this takes some time. If you want to read/copy more than one file staging is strongly recommended (background see below, command see right column).

The Windows Way

Connect to \\hsm.dmawi.de\ within windows explorer (use the right mouse button and add a network device). You can either search for shared directories or directly connect with e.g., \\hsm.dmawi.de\projects\<project>.

The Linux Way

All HSM file systems are shared and mounted automatically on most Linux clients. A simple ls /hs/projects/ should do. If you miss any directory please contact hsm-support@awi.de

Read/Write Access

The HSM system ist shared/mounted read only. To write data into the tape archive use one of the following options:

Suggestion	Command	Important Notes
Best choice	rsync -e ssh -Pauv <file|dir> <username>@hsm.dmawi.de:<destination-dir>	rsync is the most versatile way of transfering data. E.g., it allows updates with the -u option. This ensures that only new files are copied (and overwritten), existing (unchanged) files are not touched. This is important to reduce tape access. You do not want to use -a, because this would  stage all files from tape to the disk-cache for a complete file-comparison. When copying directories you need -r (recursive, already included in -a).
Fast choice	sftp filezilla	sftp provides fast way of transferring large amounts of data. Use your favourite ftp-client. However, note, that only two connections per user are allowed. If you request more, your connection will terminate. sftp uses the secure ssh-protocol and should be preferred. Use port 22 for sftp.
Do not use!	scp <file|dir> <username>@hsm.dmawi.de:<destination-dir>	scp seems convenient, but it is slightly slower when transferring data compared to ftp and/or rsync. It also just overrides existing files and no update (like rsync -u) is possible. This would also create new tape copies, you do not want to do that!!!

Note: If you have to archive many (>100 000) small (<100 MB) files this will stress the system more than necessary. Please zip or tar[.gz] your directories and upload these compressed files.

Execute commands on hsm.dmawi.de

Direct access (login) to hsm.dmawi.de is not possible. However, you can execute remote commands on hsm.dmawi.de in a restricted shell to get information about your data. E.g., you can release and stage your data if necessary. See the right column for some useful commands. They are executed with: ssh <username>@hsm.dmawi.de <command>

Change the permissions

Linux way:
As file systems are mounted read only, you have to execute a chmod on the server. As permissions for directories and files are slightly different (most files should not have the 'x') the following suggestion might be useful to restore the default permissions:

ssh hsm.dmawi.de "find /hs/<DIR> -type d -exec chmod 2775 {} \; "  # directories get  drwxrws---
ssh hsm.dmawi.de "find /hs/<DIR> -type f -exec chmod 0664 {} \;  " # files get -rw-rw----

Windows way
File permissions can be changed with filezilla or another sFTP program.

Create an ssh-key for hsm.dmawi.de

Your $HOME on hsm.dmawi.de is the standard UNIX home directory. You can use a ssh-key for hsm.dmawi.de. Execute these commands in a terminal (e.g., putty on windows):

• Execute >ssh-keygen -t rsa< on your computer/client and just press enter three times (confirmation of key location and empty passphrase).
• You can not use ssh-copy-id, because login on hsm.dmawi.de is not possible, hence you need either one of the following approaches:
  1. If you have access to another Linux Server, e.g. linsrv1.awi.de  ($HOME is identical): ssh-copy-id -i ~/.ssh/id_rsa.pub linsrv1.awi.de
  2. If you want to add your key:
     
     • scp hsm.dmawi.de:.ssh/authorized_keys /tmp/$$
     • cat ~/.ssh/*pub >> /tmp/$$
     • scp /tmp/$$ hsm.dmawi.de:.ssh/authorized_keys
  3. If this is your first ssh key:
     
     • ssh hsm.dmawi.de mkdir .ssh
     • scp ~/.ssh/id_rsa.pub hsm.dmawi.de:~/.ssh/authorized_keys
     • ssh hsm.dmawi.de "chmod 700 ~/.ssh/authorized_keys"
     
     

Note: Starting with the 7.0 release of OpenSSH, support for ssh-dsa keys has been disabled by default. You can re-enable support locally by updating your sshd_config (in /etc/ssh, /opt/local/etc/ssh, or ~/.ssh/config) with:

PubkeyAcceptedKeyTypes=+ssh-dss

Some general Information about SamFS & HSM

Principle Idea

• SamFS stands for (S)torage (a)rchive (m)anager (F)ile (S)ystem
• SamFS: is a (H)ierarchical (S)torage (M)anagement system (HSM). The HSM consists of two storage systems: The cache to speed up access, and the hierarchy. Based on a set of rules, data is stored on certain connected storage devices (tapes and maybe disks).

The Circle of Life

Archiving

• When creating a file in SamFS (e.g., by rsync, scp, ftp) the data is stored on a fast cache system (a hard drive).
• Depending on predefined policies (e.g., when the file has not been modified for a specific amount of time) the file is automatically archived on slower (and much cheaper) tapes.
• A file just created is online

Releasing

• The metadata (filename, size, ownership, permissions, etc.) of a file stay always on the cache system and are visible, but 
• when the cache system fills upl (e.g, 90% capacity) the data of large files and files that have not been touched fro some time is released.
• If the data of the file is released it is called offline.
• The user does (in the first instance) not see any difference between a online and offline file.

Staging

• When offline data is accessed the SamFS intercepts the call and automatically gathers the data from the archive media. SamFS uses informations from the metadata to find the media.
• In the meantime the reads from this file will be blocked, thus the process accessing the data blocks, too.
• When accessing more than a few files, prior staging is strongly recommended (see User commands)!

Recycling

• If the content of a file changes a new archive copy has to be produced. (You can not modify just the relevant bits on the tape.)
• The previous archive copy becomes useless (aside from having an additional backup of a previous version).
• If a file is deleted, the archive copy becomes useless, too.
• Both processes result in unused (invalid) sections on a tape. 
• Eventually only a small part of a tape contains relevant (up to date) information. The residual data is archived on other tapes, the old tape is erased and can be used for future archive copies.
• This happens by the following tasks:
  
  1. The recycler marks the tape and/or files with R.
  2. The next archiver run finds these files and starts re-archiving, the R flag of the file vanishes and a new vsn (volume serial name) for this copy is set.
  3. The recycler recognises that all files are copied somewhere else, because they have a new unique vsn. The old tape gets the status c (old candiadat) and depending on the settings in /etc/opt/SUNWsamfs/recycler.delay an atq job is scheduled for /etc/opt/SUNWsamfs/scripts/recycler.sh.

Related articles

• Page:
  Slurm-Albedo
• Page:
  System overview - albedo's hardware
• Page:
  HSM (hssrv1): Archiving (and storing) Data (hsm.dmawi.de)
• Page:
  Software - Albedo
• Page:
  Beginners guide