...
| Code Block |
|---|
| language | bash |
|---|
| title | software stack |
|---|
| linenumbers | true |
|---|
|
conda create -y -n s3 python=3.12
conda activate s3
pip install aws-shell
pip install s3cmd
conda install -y -c condo-forge s3fs boto3 python-magic pyyaml |
It is not required to install every thing as listed above. Installing only the required ones also works.
...
| Code Block |
|---|
| language | text |
|---|
| title | ~/.s3cfg |
|---|
| linenumbers | true |
|---|
|
[default]
host_base = https://hssrv2.dmawi.de:635
host_bucket = https://hssrv2.dmawi.de:635
bucket_location = bhv
access_key = HPC_user
secret_key = t1H13sOUBD/H7NuL
use_https = Yes
ca_certs_file = /Users/pasili001/Downloads/HSM_S3gw.cert.pem. # < CORRECT ME > |
Listing the buckets
| Code Block |
|---|
| language | bash |
|---|
| title | listings |
|---|
| linenumbers | true |
|---|
|
> s3cmd ls
2024-04-06 01:11 s3://testdir
> s3cmd ls s3://testdir
2024-04-06 01:11 385458 s3://testdir/tmp.csv |
upload a directory
| Code Block |
|---|
| language | bash |
|---|
| title | sync |
|---|
| linenumbers | true |
|---|
|
> s3cmd sync --stats demo-airtemp/ s3://testdir/demo-airtemp/
Done. Uploaded 5569414 bytes in 62.8 seconds, 86.61 KB/s.
Stats: Number of files transferred: 306 (5569414 bytes)
> s3cmd ls s3://testdir/demo-airtemp
DIR s3://testdir/demo-airtemp/
> s3cmd ls s3://testdir/demo-airtemp/
DIR s3://testdir/demo-airtemp/air/
DIR s3://testdir/demo-airtemp/lat/
DIR s3://testdir/demo-airtemp/lon/
DIR s3://testdir/demo-airtemp/time/
2024-04-07 15:57 307 s3://testdir/demo-airtemp/.zattrs
2024-04-07 15:57 24 s3://testdir/demo-airtemp/.zgroup
2024-04-07 15:57 3969 s3://testdir/demo-airtemp/.zmetadata |
Note: trailing forward-slash / matters in both listing the objects and as-well in transferring files ( `sync` ) to S3.