...
- You can execute (some) commands on the ScoutAM servers using ssh hssrv2 <command>
- The ScoutAM use your (standard) AWI $HOME. We suggest to create an ssh key pair with a passphrase (e.g., https://www.ssh.com/academy/ssh/keygen) there.
- The tool saminfo.sh provides more options and can be extended on request.
- hssrv2 is redirected to three different servers ( hssrv2a.dmawi.de, hssrv2b.dmawi.de and hssrv2c.dmawi.de) for load balancing. However, if you want access to a specific server, feel free to use it explicitly.
- We Because hssrv2 is redirected to different hosts (load balancing) we recommend to add these lines to your ssh-your ~/.ssh/ssh_config:
Code Block language bash title ~/.ssh/config Host hssrv2* UserKnownHostsFile /dev/null StrictHostKeyChecking no LogLevel ERROR
- The ScoutAM use your (standard) AWI $HOME. We suggest to create an ssh key pair with a passphrase (e.g., https://www.ssh.com/academy/ssh/keygen) there.
- The help of tool saminfo.sh -h shows some options and can be extended on request.
Specific taks
- We use these file variables to provide some examples. The variables are optional. You can use your filenames directly. Please note that the variable for several files have to formatted differently, depending on the preferred access method.
- Single Filesingle file
- F=/hs/store/users/2023/2023-11-30_liegmahl/home_edvs1.tgz
If you have special characters (e.g., spaces, brackets, ...) in your filename: Try to avoid them. Yes, really try to avoid them (this is no joke)! However, if are forced to use special characters it will get nasty. You have
to quote the filename and/or
protect the special characters and in the filename with "\": F="/hs/platforms/WORM/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-150529_[Record\ All]/111498_RGB/09835_041255631_500.macs".
and in addition you have to quote the ssh command correctly (see example)
- Please understand that support for this kind of "self-made problems" is very limited on our side. We officially only support recommended characters (e.g., https://www.forschungsdaten-bildung.de/dateien-organisieren)
- Directories
- D="/hs/store/users/2023/2023-11-30_liegmahl"
- Multiple Files (for saminfo.sh):
- Create a file containing all files you want to process: FL=<contains-one-file-per-line.txt> (one file per line)
- Copy the file to a specific hssrv2 server, e.g. hssrv2a: rsync $FL hssrv2a:/tmp/$FL (Hint: If $FL is located in your $HOME, there is no need to upload it )
- execute ssh hssrv2a saminfo.sh <option> -F/tmp/$FL (see examles below)
- Multiple Files (for RestAPI):
- DIR="store/users/2023/2023-11-30_liegmahl" # Note: no "/hs" here!
- DIR="store/users/2023/2023-11-30_liegmahl"
FA="/hs/$DIR/*"
- FB="/hs/$DIR/home_edvs1.tgz:/hs/$DIR/isibhv_personal.tgz:/hs/$DIR/isipd_home_pd.tgz"
FC='"$DIR/home_edvs1.tgz","$DIR/isibhv_personal.tgz","$DIR/isipd_home_pd.tgz"' # Note: "/hs/" is missing here!
- Single Filesingle file
- Some additional tasks can be done with ssh hssrv2 saminfo.sh
Access method & Task | RestAPI | ssh hssrv '...' (using CLI) | ssh hssrv 'saminfo.sh ...' (using RestAPI) | ||
---|---|---|---|---|---|
filestat (everything/all) | eval $CURL -X GET $SERV/file?path="$F" eval $CURL -X PUT --data \'{\"path\":[$FC]}\' $SERV/batchfile | sls ls -D $F $"F"' # use -E to show show | saminfo.sh -ga -f $FB"$F" | ||
online/offlinefilestat (something) | eval $CURL -X GET $SERV/file?path="$F" | jq -r '. | {path, onlineblocks, flags, error}' | saminfo.sh -o -f $F | like above, but -gs instead of -ga | ||
filestat (online/offline) | eval $CURL -X GET $SERV/file?path="$F" | jq -r .onlineblocks eval $CURL -X PUT --data \'{\"path\":[$FC]}\' $SERV/batchfile | jq -r '.infos[] | {path, onlineblocks}' | like above, but -go instead of -ga | |||
stagestaging | eval $CURL -X POST $SERV/request/stage?path="$F" eval $CURL -X PUT --data \'{\"path\":[$FC]}\' $SERV/request/batchstage | These won't work. Please use saminfo.sh instead | like above, but -s instead of -ga -s $F | ||
show scheduler state | eval $CURL -X GET $SERV/queues | -- | saminfo.sh -S | ||
|
| ||||
mkdir | -- | mkdir/hs/projects/<project>/newdir | -- | ||
release | release $F |
| saminfo.sh -f'"$F" -Crelease |
| |
quota | saminfo.sh -q | -- | |||
Count number of (online/staging) files in directory | saminfo.sh -f"$D" -Ccount |
|
File flags
The RestAPI and saminfo.sh -g[s|a] provide file information as flags. You can decode them by adding -v1 or take a look into this table:
0 | (00000000) | No flags | ||||
1 | (00000001) | Staging - File requested to stage | ||||
2 | (00000010) | StageFail - All copies failed to stage | ||||
4 | (00000100) | NoArchive - No archive copies of file should be made | ||||
8 | (00001000) | ExtCacheRequested - Policy requests extended cache | ||||
16 | (00010000) | ExtCacheDone - Extended cache copy created already | ||||
32 | (00100000) | ReleaseNever - Never release automatically |
|
|
|