Note: There is no way to get a login shell on the ScoutAM servers, but you can execute certain commands. Either via a RestAPI or using ssh or via samnfs (on Linux Systems).

hssrv2 Access

ssh hsm.dmawi.de

  • You can execute (some) commands on the ScoutAM servers using ssh hsm.dmawi.de <command>
  • hsm.dmawi.de is redirected to five different servers ( hssrv2[a-c].dmawi.de and hssrv3[a-b].dmawi.de) for high availability (HA).  However, if you want access to a specific server, feel free to use it explicitly.
  • We strongly recommend to add these lines to your ~/.ssh/ssh_config (to get HA working smoothless for you) 
    ~/.ssh/config
    Host hsm*
   UserKnownHostsFile /dev/null
   StrictHostKeyChecking no
   LogLevel ERROR
  • The ScoutAM use your (standard) AWI $HOME. We suggest to create an ssh key pair with a passphrase (e.g., https://www.ssh.com/academy/ssh/keygen) there.
  • We maintain a script for AWI users that provides (hopefully) useful options. All options are show with
     ssh hsm.dmawi.de saminfo.sh -h
    If you have suggestions for improvements, please contact us. 

RestAPI

  • These examples use  bash, curl, and jq, feel free to use the language of your choice slightly smiling face 
  • To use the RestAPI you will need a TOKEN
  • For clarity we also define the variables $CURL and $SERV for future use (see below)
TOKEN=$(curl -s -k -H 'Accept: application/json' -H 'Content-Type: application/json'  -X POST --data '{"acct":"'filestat'","pass":"'filestat'"}' https://hsm.dmawi.de:8080/v1/security/login |  jq -r '.response')
CURL='curl -s -k -H "Accept: application/json" -H "Content-Type: application/json"  -H "Authorization: Bearer $TOKEN"'
SERV='https://hsm.dmawi.de:8080/v1'
  • The token lifespan is 24 h and not client/server dependent

Specific tasks

If you have special characters (e.g., spaces, brackets, ...) in your filename: Try to avoid them. Yes, really try to avoid them (this is no joke)! However, if are forced to use special characters it will get nasty. You have

  1. to quote  the filename and/or

  2. protect the special characters and in the filename with "\": FILE="/hs/D-A/platforms/aircraft/polar6/macs/exdata/P6-244_ANT_23_24_2311300801/20231130-150529_[Record\ All]/111498_RGB/09835_041255631_500.macs".

  3. and in addition you have to quote the ssh command correctly

  4. Please understand that support for this kind of "self-made problems" is very limited on our side. We officially only support recommended  characters (e.g., https://www.forschungsdaten-bildung.de/dateien-organisieren)

samnfs (installed on linsrv2.awi.de and albedo[01])ssh hsm.dmawi.de  <command>RestAPI
ScoutFS File System

Is defined by the path (e.g., /hs/D-P)

The command eval $CURL -X GET $SERV/filesystems provides the FSID of a filesystem (They change if filesystems are recreated):

FSFSID
/hs13393638383461758966

/mnths/D-R

5426098319945997320
/mnths/D-I13385695303497597189

/mnths/D-P

11217656316633037727

Note: This documentation has not been updated for the use of multiple fs (fsid) and "batch"-commands yet ...
Variable definition

  • D="D-R/store/users/2023/2023-11-30_liegmahl"
  • FILE=/hs/$D/home_edvs1.tgz
  • DIR=/hs/$D
  • D="D-R/store/users/2023/2023-11-30_liegmahl"
  • FILE=/hs/$D/home_edvs1.tgz
  • DIR=$D                   # Note: no "/hs" here!
  • FC='"$DIR/home_edvs1.tgz","$DIR/isibhv_personal.tgz","$DIR/isipd_home_pd.tgz"'
filestat 
(everything/all)
samnfs ls /hs/D-P/projects/.../<file>
sls -D "$FILE"        # use -E to show 
sls -D "$DIR/*"       # md5 checksum

saminfo.sh -f"$FILE" -Csls
saminfo.sh -f"$DIR"  -Csls
saminfo.sh -f"$FILE" -CslsE # including 
saminfo.sh -f"$DIR"  -CslsE # md5 checksum
eval $CURL -X GET "$SERV/file?fsid=$FSID&path=$FILE"
eval $CURL -X PUT --data \'{\"path\":[$FC]}\' $SERV/batchfile
filestat
(selection)


eval $CURL -X GET "$SERV/file?fsid=$FSID&path=$F" | jq -r '. | {path, onlineblocks, flags, error}'

eval $CURL  -X PUT  --data \'{\"path\":[$FC]}\' $SERV/batchfile | jq -r '.infos[] | {path, onlineblocks, flags, error}
filestat
(online/offline)


eval $CURL -X GET "$SERV/file?fsid=$FSID&path=$F" | jq -r .onlineblocks
eval $CURL  -X PUT  --data \'{\"path\":[$FC]}\' $SERV/batchfile | jq -r '.infos[] | {path, onlineblocks}'
stage
samnfs stage /hs/D-P/projects/.../<file>
saminfo.sh -f"$FILE" -Cstage
saminfo.sh -f"$DDIR" -Cstage
eval $CURL -X POST "$SERV/request/stage?fsid=$FSID&path=$F"
eval $CURL -X PUT  --data \'{\"path\":[$FC]}\' $SERV/request/batchstage
show scheduler state

saminfo.sh -S
eval $CURL -X GET $SERV/queues
online/offline information

saminfo.sh -Cc -f$DIR

mkdir

mkdir /hs/D-P/projects/<project>/newdir

release
samnfs release /hs/D-P/projects/.../<file>
saminfo.sh -f"$FILE" -Crelease
saminfo.sh -f"$DIR" -Crelease

quota
samnfs quota
saminfo.sh -q

Count number of
(online/staging)
files in directory

saminfo.sh -f"$DIR" -Ccount

ssh-Quoting

You can provide placeholders if (and only if) you quote correctly. E.g.,

ssh hsm.dmawi.de "saminfo.sh -Cc -f '/hs/D-A/platforms/vessel/polarstern/parasound_p70/exdata/PS138/PHF2309162*' "

Multiple files  with ssh

(Note: This is not implemented for multiple ScoutFS, yet. If you think you really need this, contact us and if we have time left, we start implementing this.) 

You can process multiple files with ssh:

  1. Create a file containing all files you want to process: FL=<contains-one-file-per-line-with-absolute-path.txt>
  2. Copy the file to a specific hssrv2 server, e.g.  rsync $FL hssrv2a:/tmp/$FL (Hint: If $FL is located in your $HOME, there is no need to upload it (wink))
  3. execute ssh hssrv2a saminfo.sh <option> -F/tmp/$FL    Options may be:
    -ga    # filestat
    -gs    # filestat selection
    -go   # filestat online/offline
    -s      # stage

File flags

samnfs provides these flags:

ScoutAM file flags 
List files and directories with archive information.

Only supports long listing with the following fields:

  File mode
  Link count
  User name or ID
  Group name or ID
  File size
  Modification timestamp
  ScoutAM archive file flags
  Valid archive copy count
  File name

ScoutAM file flags:

  Archive flags:
    a----   File is marked as archdone
    n----   File is marked as no archive
    -----   File is not marked as archdone

  Cache flags:
    -o---   File is online
    -n---   File is marked as release never
    -----   File is offline

  Stating flags:
    --s--   File is staging
    --X--   File is marked as stage failed
    -----   File is not staging

  Retention flags:
    ---r-   File has retention set
    -----   File does not have retention set

  Checksum flags:
    ----n   File has checksum set but has not been verified
    ----v   File has checksum set and has been verified
    ----X   File has checksum set but verified invalid
    -----   File does not have a user checksum

Usage:
  samnfs ls [flags] files...

Flags:
  -a, --all        Print all files including hidden files
  -C, --checksum   Print user/VSM checksums if it exists
  -h, --help       help for ls
      --human      Print sizes in human readable format

The RestAPI and saminfo.sh -g[s|a] provide file information as flags. You can decode them by adding -v1 to saminfo.sh or take a look into this table:

0

(00000000)No flags
1(00000001)Staging - File requested to stage
2(00000010)StageFail - All copies failed to stage
4(00000100)NoArchive - No archive copies of file should be made
8(00001000)ExtCacheRequested - Policy requests extended cache
16(00010000)ExtCacheDone - Extended cache copy created already
32(00100000)ReleaseNever - Never release automatically


  • No labels