Versions Compared

Old Version 4

changes.mady.by.user Malte Thoma

Saved on

compared with

New Version Current

changes.mady.by.user Malte Thoma

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • These examples use  bash, curl, and jq, feel free to use the language of your choice slightly smiling face 
  • To use the RestAPI you will need a TOKEN
  • For clarity we also define the variable variables $CURL and $SERV for future use (see below)
Code Block
languagebash
TOKEN=$(curl -s -k -H 'Accept: application/json' -H 'Content-Type: application/json'  -X POST --data '{"acct":"'filestat'","pass":"'filestat'"}' https://hssrv2.dmawi.de:8080/v1/security/login |  jq -r '.response')

CURL='curl -s -k -H "Accept: application/json" -H "Content-Type: application/json"  -H "Authorization: Bearer $TOKEN"'
SERV='https://hssrv2.dmawi.de:8080/v1'

...

  • You can execute (some) commands on the ScoutAM servers using ssh hssrv2 <command>
  • hssrv2 is redirected to three different servers ( hssrv2a.dmawi.de, hssrv2b.dmawi.de and hssrv2c.dmawi.de) for load balancing.  However, if you want access to a specific server, feel free to use it explicitly.
  • We recommend to add these lines to your ~/.ssh/ssh_config: 
    Code Block
    languagebash
    title~/.ssh/config
    Host hssrv2*
   UserKnownHostsFile /dev/null
   StrictHostKeyChecking no
   LogLevel ERROR
  • The ScoutAM use your (standard) AWI $HOME. We suggest to create an ssh key pair with a passphrase (e.g., https://www.ssh.com/academy/ssh/keygen) there.
  • The help of tool saminfo.sh -h shows some options and can be extended on request. 

Specific taks

filestat (e.g. online/offline)

  • We use these variables for testingto provide some examples. The variables are optional. You can use your filenames directly. Please note that the variable for several files have to formatted differently, depending on the preferred access method. 
      single file:
      • Single File
        • F=/hs/store/users/2023/2023-11-30_liegmahl/home_edvs1.tgz
      multiple files:

        • If you have special characters (e.g., spaces, brackets, ...) in your filename: Try to avoid them. Yes, really try to avoid them (this is no joke)! However, if are forced to use special characters it will get nasty. You have

          1. to quote  the filename and/or

          2. protect the special characters and in the filename with "\": F="/hs/platforms/WORM/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-150529_[Record\ All]/111498_RGB/09835_041255631_500.macs".

          3. and in addition you have to quote the ssh command correctly (see example)

          4. Please understand that support for this kind of "self-made problems" is very limited on our side. We officially only support recommended  characters (e.g., https://www.forschungsdaten-bildung.de/dateien-organisieren)
      • Directories
        • D="/hs/store/users/2023/2023-11-30_liegmahl"
      • Multiple Files (for saminfo.sh):
        1. Create a file containing all files you want to process: FL=<contains-one-file-per-line.txt> (one file per line)
        2. Copy the file to a specific hssrv2 server, e.g. hssrv2a: rsync $FL hssrv2a:/tmp/$FL (Hint: If $FL is located in your $HOME, there is no need to upload it (wink))
        3. execute ssh hssrv2a saminfo.sh <option> -F/tmp/$FL   (see examles below)
      • Multiple Files (for RestAPI):
        • DIR="store/users/2023/2023-11-30_liegmahl"                       # Note: no "/hs" here!
      • DIR="store/users/2023/2023-11-30_liegmahl"
      • FA="/hs/$DIR/*"
      • FB="/hs/$DIR/home_edvs1.tgz:/hs/$DIR/isibhv_personal.tgz:/hs/$DIR/isipd_home_pd.tgz"
        • FC='"$DIR/home_edvs1.tgz","$DIR/isibhv_personal.tgz","$DIR/isipd_home_pd.tgz"'
        # Note: "/hs" is missing !
    • Some additional tasks can be done with ssh hssrv2 saminfo.sh 
    Access method 
    & Task    		RestAPIssh hssrv '...'
    (using CLI)    		ssh hssrv 'saminfo.sh ...'
    (using RestAPI)
    filestat 
    (everything/all)
    		eval $CURL -X GET $SERV/file?path="$F"
    eval $CURL  -X PUT  --data \'{\"path\":[$FC]}\' $SERV/batchfile
    ls -D $"F"'        # use -E to show 
    sls -D "$D/*"    # md5 checksum

    saminfo.sh -f"$F
    sls -D $FA" -Csls
    saminfo.sh -f"$D" -Csls
    saminfo.sh -ga -f"$F"
    saminfo.sh -ga -f $FB"$D"
    saminfo.sh -ga -F/tmp/$FL
    online/offlinefilestat
    (something)
    eval $CURL -X GET $SERV/file?path="$F" | jq -r '. | {path, onlineblocks, flags, error}'
    eval $CURL  -X PUT  --data \'{\"path\":[$FC]}\' $SERV/batchfile | jq -r '.infos[] | {path, onlineblocks, flags, error}
    '
    saminfo.sh -o -f $F
    saminfo.sh -o -f $FB

    like above, but -gs instead of -ga
    filestat 
    (online/offline)
    		eval $CURL -X GET $SERV/file?path="$F" | jq -r .onlineblocks
    eval $CURL  -X PUT  --data \'{\"path\":[$FC]}\' $SERV/batchfile | jq -r '.infos[] | {path, onlineblocks}'

    like above, but -go instead of -ga
    stagestaging
    		eval $CURL -X POST $SERV/request/stage?path="$F"
    eval $CURL -X PUT  --data \'{\"path\":[$FC]}\' $SERV/request/batchstagestage $F
    stage $FF
    saminfo.sh -f'"$F" -Cstage
    saminfo.sh -f'"$D" -Cstage
    like above, but -s instead of -ga

    show scheduler state
    		eval $CURL -X GET $SERV/queues
     --
    saminfo.sh -S


     
     
    mkdir
    		--
    mkdir/hs/projects/<project>/newdir
    --
    release

    saminfo.sh -s $F
    f'"$F" -Crelease
    saminfo.sh -f'"$D" -Crelease
     
    quota

     saminfo.sh -q
    --
    Count number of 
    (online/staging)
    files in directory

     saminfo.sh -f"$D" -Ccount
     
    File flags
    The RestAPI and saminfo.sh -g[s|a] provide file information as flags. You can decode them by adding -v1 or take a look into this table:
    0
    		(00000000)No flags
    1(00000001)Staging - File requested to stage
    2(00000010)StageFail - All copies failed to stage
    4(00000100)NoArchive - No archive copies of file should be made
    8(00001000)ExtCacheRequested - Policy requests extended cache
    16(00010000)ExtCacheDone - Extended cache copy created already
    32(00100000)ReleaseNever - Never release automaticallys $FB