...
Note: If you have to archive many (>100 000) small (<100 MB) files this will stress the system more than necessary. Please zip or tar[.gz] your directories and upload these compressed files.
Example: Copy a lot of data from HSM to HPC
Assuming you want to copy a lot of data from "/hs/platforms/WORM/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_[Record All]/111498_RGB" (note the special characters you should avoid on all costs 
)
- Check how many files are online:
Code Block language bash ssh hssrv2 "saminfo.sh -Cc -f '/hs/platforms/WORM/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_[Record All]/111498_RGB'" Checking >/hs/platforms/WORM/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_[Record All]/111498_RGB< Total files: ... 12083 Online files: ... 7637 Staging files: ... 4299 Size: ... 857G - Stage your data (only needed once, in the output shown before this was done some time ago):
Code Block language bash ssh hssrv2 "saminfo.sh -Cstage -f '/hs/platforms/WORM/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_[Record All]/111498_RGB'" - You can start copying already staged files whenever you want:
Code Block language bash ssh hssrv2 "samcli file find /hs/platforms/WORM/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_\[Record\ All\]/111498_RGB/ --online " > /tmp/files-online.txt rsync -Pauv --no-R --chmod 664 --files-from=/tmp/files-online.txt hssrv2:/ . You can repeat step 3 whenever you feel it is necessary and finally copy all remaining files (when thy are finally online) with
Code Block language bash rsync -Pauv --chmod 664 hssrv2:/hs/platforms/WORM/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-141433_\[Record\ All\]/111498_RGB/ .
Object Storage (S3)
- ScoutAM provides a S3-gateway to access data.
- Currently this in the testing phase.
- We consider/plan to provide S3 storage via eResources in he future (Speculation: Q4/2024)
- Only TLS connections are possible. Please use this certificate: HSM_S3gw.cert.pem