...
| Code Block |
|---|
| language | bash |
|---|
| title | software stack |
|---|
| linenumbers | true |
|---|
|
conda create -y -n s3 python=3.12
conda activate s3
pip install aws-shell
pip install s3cmd
conda install -y -c condo-forge s3fs boto3 pyyaml |
It is not required to install every thing as listed above. Installing only the required ones also works.
credentials
Lets say the following information is provided by the system administrator
| Code Block |
|---|
| language | text |
|---|
| title | credentials |
|---|
| linenumbers | true |
|---|
|
URL:PORT => https://hssrv2.dmawi.de:635
region/location => bhv
ACCESS_KEY => HPC_user
SECRET_KEY => t1H13sOUBD/H7NuL
CERTS_FILE => https://spaces.awi.de/download/attachments/494210152/HSM_S3gw.cert.pem |
These credentials needs to be adapted for each of the clients.
aws
Use aws configure to adapt the credentials to this tool or create the following files.
| Code Block |
|---|
| language | text |
|---|
| title | ~/.aws/credentials |
|---|
| linenumbers | true |
|---|
|
[default]
aws_access_key_id=HPC_user
aws_secret_access_key=t1H13sOUBD/H7NuL |
| Code Block |
|---|
| language | text |
|---|
| title | ~/.aws/config |
|---|
| linenumbers | true |
|---|
|
[default]
region = bhv
endpoint_url = https://hssrv2.dmawi.de:635
ca_bundle = ${HOME}/Downloads/HSM_S3gw.cert.pem |
Listing the buckets
| Code Block |
|---|
| language | bash |
|---|
| title | listings |
|---|
| linenumbers | true |
|---|
|
> aws s3 ls
2024-04-06 01:11:30 testdir
> aws s3 ls s3://testdir
2024-04-06 01:11:30 385458 tmp.csv |
s3cmd
s3cmd is a free command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol.
s3cmd look for credentials at ${HOME}/.s3cfg
create the config file as follows
| Code Block |
|---|
| language | text |
|---|
| title | ~/.s3cfg |
|---|
| linenumbers | true |
|---|
|
[default]
host_base = https://hssrv2.dmawi.de:635
host_bucket = https://hssrv2.dmawi.de:635
bucket_location = bhv
access_key = HPC_user
secret_key = t1H13sOUBD/H7NuL
use_https = Yes
#check_ssl_certificate = False |