Versions Compared

Old Version 1

changes.mady.by.user Fabian Harms

Saved on

compared with

New Version 2

changes.mady.by.user Fabian Harms

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Your Data is archived on a tape, so it will take a while to get it back.
  2. You have a quota in your personal user space (domain C) of 1 TB (10 TB) online (offline) and 200000 files. (You can check your quota with ssh hssrv2 saminfo.sh -q)
  3. User data will be deleted six months after a user leaves AWI. But you can move your data from the user space into the project area (domain B), where it will be stored for a much longer period (domain B) or even permanently (domain A).
Three domains of archiving at AWI
Image Modified


DomainPurpose
APermanent archive for irrecoverable data. Metadata is needed for this domain. Please contact Stefanie Schumacher for more information about Pangaea.
BLong term project data with predefined life time. A project can be created with eResources https://cloud.awi.de/#/projects A sticky bit in the project area protects files from being deleted by anyone other than the admin or the owner. The setgid on directories ensures that new (sub-)directories belong to the UNIX-group automatically.
CEvery user get a personal user area when he gets an account on hssrv2. However, user data will be deleted after user leaves AWI.



DomainFile SystemsTape ArchiveDisk Archive
How to apply

A

/hs/usera


Yes


Yes


Archive of individual Projects



/hs/useroYesYesPangaea
B


/hs/bsys

Yes

Yes

Biological science

Please use eResources https://cloud.awi.de/#/projects to create a project.



/hs/csysYesYesClimate science
/hs/gsysYesYesGeophysics science
/hs/techYesYesTechnical science

/hs/potsdamYesNo
C

/hs/userc



Yes

No

User data



To apply for a HSM account you need an AWI-Unix account first.

/hs/usermYesNoUser data

A disk archive is available for specific files at domains A and B. This allows a fast access of offline files. The availability of this disk archive depends on the actual resources/usage and might change.

...

You need an HSM-account, if you want to write data on the tape system. After you have been informed via email, you can archive your files with the following methods:

SuggestionCommandImportant Notes
Best choice :-)

rsync -e ssh -uvP[r] <file|dir> <username>@hssrv2.awi.de:<destination-dir>

rsync is the most versatile way of transfering data. E.g., it allows updates with the -u option. This ensures that only new files are copied (and overwritten), existing (unchanged) files are not touched. This is important to reduce tape access. You do not want to use -a, because this would  stage all files from tape to the disk-cache for a complete file-comparison.

When copying directories you need -r (recursive).

Fast choice :-)sftp/ftp[s]ftp provides the fastest way of transferring large amounts of data. Use your favourite ftp-client. However, note, that only two connections per user are allowed. If you request more, your connection will terminate. sftp uses the secure ssh-protocol and should be preferred.
Do not use! :-(scp <file|dir> <username>@hssrv2.awi.de:<destination-dir>scp seems convenient, but it is slightly slower when transferring data compared to ftp and/or rsync. It also just overrides existing files and no update (like rsync -u) is possible. This would also create new tape copies, you do not want to do that!!!

Note: If you have to archive many (>100 000) small (<100 MB) files this will stress the system more than necessary. Please zip or tar your directories and upload these compressed files.

...