Suggestions/good to know

  1. Storing millions of files in one directory slows down any filesystem. Therefore we suggest to limit the number of files whenever possible (per design, workflow, zip, tar, ...)
  2. We suggest to limit the size of each individual file to a maximum of 500 GB per file. E.g. with
    tar -cPf - $INDIRECTORY | pigz -c | split -a3 -d -b500GB - $OUTFILE.
  3. If you access(read) more than one file from the HSM, please stage all files you plan to access beforehand at once! This will decrease the overall access time significantly by reducing the necessary robotic accesses to a minimum. However, please limit yourself to access only about a dozen TB at once. 

Block/File Storage (POSIX, smb, nfs)

Read only Access

The Windows Way (smb/cifs)

Connect to \\hsm.dmawi.de\ within windows explorer (use the right mouse button and add a network device). You can either search for shared directories or directly connect with e.g., \\hsm.dmawi.de\projects\<project>.

The Linux Way

nfs

The HSM file systems are shared and mounted automatically on most Linux server and clients (but not on laptops). A simple ls /hs/D-P/projects/ should do. If you should miss a directory please contact hsm-support@awi.de

cifs/smb

On laptops you can mount HSM directories via smb. You can use any file manager (thunar, nautilus, ...) or the commad line:

sudo mount    -t cifs -o username=$USER@dmawi.de          //hsm/Projects          /mnt  # short form
sudo mount -v -t cifs -o username=$USER@dmawi.de,vers=3.0 //hsm.dmawi.de/Projects /mnt  # long form


# "sometimes" these work, too:
# kinit $USER
# sudo mount -t cifs -o sec=krb5 //hsm.dmawi.de/Projects /mnt # short form
# sudo mount -t cifs -o username=$USER,domain=DMAWI,workgroup=DMAWI.DE,password=$PW,sec=krb5,vers=3.11 //hsm.dmawi.de/Projects /mnt  # long form

Read/Write Access

The HSM system ist shared/exported/mounted read only. To write data into the tape archive use one of the following options:

SuggestionCommandImportant Notes
(big grin) Best choice

rsync -e ssh -Pauv <file|dir> <username>@hsm.dmawi.de:<destination-dir>

rsync is the most versatile way of transferring data. E.g., it allows updates with the -u option. This ensures that only new files are copied (and overwritten), existing (unchanged) files are not touched. This is important to avoid tape waste.
Note: You do not want to use -c, because this would  stage all files from tape to the disk-cache for a complete file-comparison.

When copying directories you need -r (recursive, already included in -a).

(big grin) Fast choicesftp
filezilla
sftp provides fast way of transferring large amounts of data. Use your favorite ftp-client. However, note, that only two connections per user (per HSM server) are allowed. If you request more, your connection will terminate. sftp uses the secure ssh-protocol and should be preferred. Use port 22 for sftp.
(minus) Do not use!scp <file|dir> <username>@hsm.dmawi.de:<destination-dir>scp seems convenient, but it is slightly slower when transferring data compared to ftp and/or rsync. It also just overrides existing files and no update (like rsync -u) is possible. This would also create new tape copies, you do not want to do that!!!

Note: If you have to archive many (>100 000) small (<100 MB) files this will stress the system more than necessary. Please zip or tar[.gz] your directories and upload these compressed files.

Example: Copy a lot of data from HSM to HPC

Assuming you want to copy a lot of data from "/hs/D-A/platforms/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_*/111498_RGB/1*" (note the the asterisk is allowed (smile)  )

  1. Check how many files are online:
    ssh hsm.dmawi.de "saminfo.sh -Cc -f '/hs/D-A/platforms/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_*/111498_RGB/1*'"
Checking >/hs/D-A/platforms/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_*/111498_RGB/1*<
   Total files:   ... 2084
   Online files:  ... 6
   Offline files: ... 2076
   Staging files: ... 100
   Total Size:    ... 147.794 GB
  2. Stage your data (only needed once, in the output shown before this was done some time ago):  
    ssh hsm.dmawi.de "saminfo.sh -Cstage -f '/hs/D-A/platforms/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_*/111498_RGB/1*'"
  3. You can start copying already staged files whenever you want:
    ssh hsm.dmawi.de "samcli file find /hs/D-A/platforms/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_*/111498_RGB/1* --online " > /tmp/files-online.txt
rsync -Pauv --no-R --chmod 664 --files-from=/tmp/files-online.txt hsm.dmawi.de:/ .

  4. You can repeat step 3 whenever you feel it is necessary and finally copy all remaining files (when thy are finally online) with

    rsync -Pauv --chmod 664 hsm.dmawi.de:/hs/D-A/platforms/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_*/111498_RGB/1* .


Object Storage (S3)