Note: There is no way to get a login shell on the ScoutAM servers, but you can execute certain commands. Either via a RestAPI or using ssh.

hssrv2 Access

ssh hsm.dmawi.de

  • You can execute (some) commands on the ScoutAM servers using ssh hsm.dmawi.de <command>
  • hsm.dmawi.de is redirected to three different servers ( hssrv2a.dmawi.de, hssrv2b.dmawi.de and hssrv2c.dmawi.de) for load balancing.  However, if you want access to a specific server, feel free to use it explicitly.
  • We recommend to add these lines to your ~/.ssh/ssh_config: 
    ~/.ssh/config
    Host hsm*
   UserKnownHostsFile /dev/null
   StrictHostKeyChecking no
   LogLevel ERROR
  • The ScoutAM use your (standard) AWI $HOME. We suggest to create an ssh key pair with a passphrase (e.g., https://www.ssh.com/academy/ssh/keygen) there.
  • We maintain a script for AWI users that provides (hopefully) useful options. All options are show with
     ssh hsm.dmawi.de saminfo.sh -h
    If you have suggestions for improvements, please contact us. 

RestAPI

  • These examples use  bash, curl, and jq, feel free to use the language of your choice slightly smiling face 
  • To use the RestAPI you will need a TOKEN
  • For clarity we also define the variables $CURL and $SERV for future use (see below)
TOKEN=$(curl -s -k -H 'Accept: application/json' -H 'Content-Type: application/json'  -X POST --data '{"acct":"'filestat'","pass":"'filestat'"}' https://hsm.dmawi.de:8080/v1/security/login |  jq -r '.response')
CURL='curl -s -k -H "Accept: application/json" -H "Content-Type: application/json"  -H "Authorization: Bearer $TOKEN"'
SERV='https://hsm.dmawi.de:8080/v1'
  • The token lifespan is 24 h and not client/server dependent

Specific tasks

If you have special characters (e.g., spaces, brackets, ...) in your filename: Try to avoid them. Yes, really try to avoid them (this is no joke)! However, if are forced to use special characters it will get nasty. You have

  1. to quote  the filename and/or

  2. protect the special characters and in the filename with "\": FILE="/hs/D-A/platforms/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-150529_[Record\ All]/111498_RGB/09835_041255631_500.macs".

  3. and in addition you have to quote the ssh command correctly

  4. Please understand that support for this kind of "self-made problems" is very limited on our side. We officially only support recommended  characters (e.g., https://www.forschungsdaten-bildung.de/dateien-organisieren)

ssh hsm.dmawi.de  <command>RestAPI
ScoutFS File System
Is defined by the path (e.g., /hs/D-P)

The command eval $CURL -X GET $SERV/filesystems provides the FSID of a filesystem (They change if filesystems are recreated):

FSFSID
/hs13393638383461758966

/mnths/D-R

5426098319945997320
/mnths/D-I13385695303497597189

/mnths/D-P

11217656316633037727

Note: This documentation has not been updated for the use of multiple fs (fsid) and "batch"-commands yet ...
Variable definition
  • D="D-R/store/users/2023/2023-11-30_liegmahl"
  • FILE=/hs/$D/home_edvs1.tgz
  • DIR=/hs/$D
  • D="D-R/store/users/2023/2023-11-30_liegmahl"
  • FILE=/hs/$D/home_edvs1.tgz
  • DIR=$D                   # Note: no "/hs" here!
  • FC='"$DIR/home_edvs1.tgz","$DIR/isibhv_personal.tgz","$DIR/isipd_home_pd.tgz"'
filestat 
(everything/all)
sls -D "$FILE"        # use -E to show 
sls -D "$DIR/*"       # md5 checksum

saminfo.sh -f"$FILE" -Csls
saminfo.sh -f"$DIR"  -Csls
eval $CURL -X GET "$SERV/file?fsid=$FSID&path=$FILE"
eval $CURL -X PUT --data \'{\"path\":[$FC]}\' $SERV/batchfile
filestat
(selection)

eval $CURL -X GET "$SERV/file?fsid=$FSID&path=$F" | jq -r '. | {path, onlineblocks, flags, error}'

eval $CURL  -X PUT  --data \'{\"path\":[$FC]}\' $SERV/batchfile | jq -r '.infos[] | {path, onlineblocks, flags, error}
filestat
(online/offline)

eval $CURL -X GET "$SERV/file?fsid=$FSID&path=$F" | jq -r .onlineblocks
eval $CURL  -X PUT  --data \'{\"path\":[$FC]}\' $SERV/batchfile | jq -r '.infos[] | {path, onlineblocks}'
stage
saminfo.sh -f"$FILE" -Cstage
saminfo.sh -f"$DDIR" -Cstage
eval $CURL -X POST "$SERV/request/stage?fsid=$FSID&path=$F"
eval $CURL -X PUT  --data \'{\"path\":[$FC]}\' $SERV/request/batchstage
show scheduler state
saminfo.sh -S
eval $CURL -X GET $SERV/queues
online/offline information
saminfo.sh -Cc -f$DIR

mkdir
mkdir /hs/D-P/projects/<project>/newdir

release
saminfo.sh -f'"$FILE" -Crelease
saminfo.sh -f'"$DIR" -Crelease

quota
saminfo.sh -q

Count number of
(online/staging)
files in directory
saminfo.sh -f"$DIR" -Ccount

ssh-Quoting

You can provide placeholders if (and only if) you quote correctly. E.g.,

ssh hsm.dmawi.de "saminfo.sh -Cc -f '/hs/D-A/platforms/vessel/polarstern/parasound_p70/exdata/PS138/PHF2309162*' "

Multiple files  with ssh

(Note: This is not implemented for multiple ScoutFS, yet. If you think you really need this, contact us and if we have time left, we start implementing this.) 

You can process multiple files with ssh:

  1. Create a file containing all files you want to process: FL=<contains-one-file-per-line-with-absolute-path.txt>
  2. Copy the file to a specific hssrv2 server, e.g.  rsync $FL hssrv2a:/tmp/$FL (Hint: If $FL is located in your $HOME, there is no need to upload it (wink))
  3. execute ssh hssrv2a saminfo.sh <option> -F/tmp/$FL    Options may be:
    -ga    # filestat
    -gs    # filestat selection
    -go   # filestat online/offline
    -s      # stage

File flags

The RestAPI and saminfo.sh -g[s|a] provide file information as flags. You can decode them by adding -v1 to saminfo.sh or take a look into this table:

0

(00000000)No flags
1(00000001)Staging - File requested to stage
2(00000010)StageFail - All copies failed to stage
4(00000100)NoArchive - No archive copies of file should be made
8(00001000)ExtCacheRequested - Policy requests extended cache
16(00010000)ExtCacheDone - Extended cache copy created already
32(00100000)ReleaseNever - Never release automatically


  • No labels