Please note: Although these Information about the usage of S3 are correct, the S3-gateway@AWI is still under construction and not available for users, yet. However, if you have use cases you may contact Pavan Kumar Siligam . He collects use cases.
Access Variables
Your ACCESS_KEY and SECRET_KEY are issued via eResources — contact your group administrator if you do not have them. These names are used consistently throughout this page.
| Variable | Value | Used by application | Description |
|---|---|---|---|
ENDPOINT | https://hsm.dmawi.de | aws, s3fs-fuse | S3 service URL |
ENDPOINT_ALT | https://hssrv2.dmawi.de | s3cmd, s3fs, botocore | S3 service URL |
PORT | 7071 | all | Service port |
REGION | bremerhaven | all | Storage region |
ACCESS_KEY | <your key from eResources> | all | S3 access key identifier |
SECRET_KEY | <your secret from eResources> | all | S3 secret access key |
CA_BUNDLE | HSM_S3gw.cert.pem | all | TLS certificate |
Download CA_BUNDLE before using the tools
Set restrictive permissions on all credential files: `chmod 600 <credentials-file>`
Which tool suits you?
| I want to... | Use |
|---|---|
| Transfer files from the command line | s3cmd / aws |
| Sync a directory (like rsync) | s3cmd / aws |
| Browse the bucket like a local folder | s3fs-fuse |
| Access S3 from a Python script | s3fs |
| Fine-grained control / build a library | botocore |
| Store numpy arrays or zarr datasets in S3 | s3fs + zarr |
Install only what you need:
conda create -y -n s3 python=3.12 && conda activate s3 pip install s3cmd aws-shell conda install -y -c conda-forge s3fs-fuse s3fs boto3 pyyaml
S3 Concepts
It is good to be aware of some terminology as they come up with the tools we use.
PREFIX
|----------|
s3://pavan-test/demo-airtemp/lat/.zattrs
|--------| |----------------------|
BUCKET OBJECT
When listing the contents of a bucket the trailing `/` matters — `s3://testdir/demo-airtemp` and `s3://testdir/demo-airtemp/` behave differently.
s3cmd – command-line transfers
Best for: interactive use, upload/download, directory sync (rsync/ftp style).
credentials file: ~/.s3cfg
[default] host_base = $ENDPOINT_ALT:$PORT host_bucket = $ENDPOINT_ALT:$PORT bucket_location = $REGION access_key = $ACCESS_KEY secret_key = $SECRET_KEY use_https = Yes ca_certs_file = $CA_BUNDLE
Example usage of this command
s3cmd ls # list buckets s3cmd ls s3://testdir/ # list objects s3cmd du -H s3://testdir/ # disk usage s3cmd get s3://testdir/file.csv ./file.csv # download s3cmd put ./file.csv s3://testdir/file.csv # upload s3cmd sync local-dir/ s3://testdir/remote-dir/ # sync directory to S3 s3cmd sync s3://testdir/remote-dir/ local-dir/ # sync S3 to local
aws – command-line transfers
Best for: scripting, automation, AWS-compatible workflows.
The aws tool splits its configuration across two files: ~/.aws/credentials holds your identity (access key and secret), while ~/.aws/config holds connection settings (endpoint, region, certificate). This separation allows credentials to be managed independently from environment-specific configuration.
# [file: ~/.aws/credentials] [default] aws_access_key_id = $ACCESS_KEY aws_secret_access_key = $SECRET_KEY # [file: ~/.aws/config] [default] region = $REGION endpoint_url = $ENDPOINT:$PORT ca_bundle = $CA_BUNDLE
Example usage
aws s3 ls # list buckets aws s3 ls s3://testdir/ # list objects aws s3 cp s3://testdir/file.csv ./file.csv # download aws s3 cp ./file.csv s3://testdir/file.csv # upload aws s3 sync local-dir/ s3://testdir/remote-dir/ # sync directory to S3 aws s3 sync s3://testdir/remote-dir/ local-dir/ # sync S3 to local
s3fs-fuse – mount as filesystem
Best for: browsing or accessing bucket contents without changing its contents.
# [file: ~/.passwd-s3fs] <- this filename is relaxed. meaning name it anything you like. $ACCESS_KEY:$SECRET_KEY
Example usage
mkdir -p /tmp/mybucket
s3fs testdir /tmp/mybucket \
-o url=$ENDPOINT:$PORT \
-o passwd_file=~/.passwd-s3fs \
-o cacertfile=$CA_BUNDLE
ls /tmp/mybucket/ # browse like a local folder
fusermount -u /tmp/mybucket # unmount when done
s3fs – Python filesystem interface
Best for: Python scripts that need to list, read, or write files in S3.
credentials file: ~/.s3fs
key: $ACCESS_KEY secret: $SECRET_KEY client_kwargs: endpoint_url: $ENDPOINT_ALT:$PORT verify: $CA_BUNDLE region_name: $REGION
Utility function to read the config file
import os, yaml, s3fs
def get_fs():
with open(os.path.expanduser("~/.s3fs")) as fid:
return s3fs.S3FileSystem(**yaml.safe_load(fid))
Example usage
fs = get_fs()
fs.ls('testdir/') # list objects
fs.get('testdir/file.csv', './file.csv') # download
fs.put('./file.csv', 'testdir/file.csv') # upload
# read directly without downloading to disk
with fs.open('testdir/file.csv', 'rb') as f:
data = f.read()
Full API: https://s3fs.readthedocs.io/en/latest/api.html
Botocore – low level Python interface
Best for: library builders, fine-grained control, or when you need full response metadata.
credentials file: ~/.s3fs_boto File naming is relaxed. meaning name it as you like. Note: the contents are in yaml format.
service_name: s3 aws_access_key_id: $ACCESS_KEY aws_secret_access_key: $SECRET_KEY endpoint_url: $ENDPOINT_ALT:$PORT region_name: $REGION verify: $CA_BUNDLE
Utility function to read the config file
import os, yaml, boto3
def get_connection():
with open(os.path.expanduser("~/.s3fs_boto")) as fid:
return boto3.client(**yaml.safe_load(fid))
common operations
conn = get_connection()
conn.list_buckets() # list buckets
conn.list_objects(Bucket='testdir') # list objects (max 1000)
conn.download_file('testdir', 'file.csv', 'file.csv') # download
conn.upload_file('file.csv', 'testdir', 'file.csv') # upload
Unlike s3fs, botocore returns a detailed response dictionary for every call. For example, `list_objects` returns not just filenames but also size, last-modified timestamp, ETag (achecksum), and HTTP response metadata — useful when you need to verify transfers, detect changes, or build tooling on top of S3. For everyday file access this level of detail is unnecessary, but it becomes valuable when writing scripts that need to act on object state.
Listing buckets and objects
>>> conn = get_connection()
>>> # Listing buckets
>>> print(conn.list_buckets())
{'Buckets': [{'CreationDate': datetime.datetime(2024, 4, 7, 15, 57, 46, 944296, tzinfo=tzoffset(None, 7200)),
'Name': 'testdir'}],
'Owner': {'DisplayName': '', 'ID': '$GRP'},
'ResponseMetadata': {'HTTPHeaders': {'connection': 'close',
'content-length': '315',
'content-type': 'application/xml',
'date': 'Sun, 07 Apr 2024 21:50:03 GMT',
'server': 'VERSITYGW'},
'HTTPStatusCode': 200,
'RetryAttempts': 0}}
>>>
>>> # filtering down the results just to show the bucket names
>>> for bucket in conn.list_buckets().get('Buckets'):
... print(bucket['Name'])
...
'testdir'
>>> # Listing objects
>>> objs = conn.list_objects(Bucket='testdir')
>>> print(obj)
{'Delimiter': '',
'EncodingType': '',
'IsTruncated': False,
'Marker': '',
'MaxKeys': 1000,
'Name': 'testdir',
'NextMarker': '',
'Prefix': '',
'ResponseMetadata': {'HTTPHeaders': {'connection': 'close',
'content-length': '67702',
'content-type': 'application/xml',
'date': 'Sun, 07 Apr 2024 21:58:15 GMT',
'server': 'VERSITYGW'},
'HTTPStatusCode': 200,
'RetryAttempts': 0},
'Contents': [{'ETag': '5f0137574247761b438aa508333f487d',
'Key': 'tmp.csv',
'LastModified': datetime.datetime(2024, 4, 6, 1, 11, 30, 890787, tzinfo=tzoffset(None, 7200)),
'Size': 385458,
'StorageClass': 'STANDARD'},
...
...
{'ETag': '7c6e83fce9aa546ec903ca93f036a2fd',
'Key': 'demo-airtemp/time/0',
'LastModified': datetime.datetime(2024, 4, 7, 15, 58, 49, 630102, tzinfo=tzoffset(None, 7200)),
'Size': 2549,
'StorageClass': 'STANDARD'}]}
To get just the filenames:
objs = conn.list_objects(Bucket='testdir')
names = [o['Key'] for o in objs.get('Contents', [])]
# ['file.csv', 'demo-airtemp/lat/.zattrs', ...]
For buckets with more than 1000 objects, listing is paginated via `IsTruncated` / `NextMarker`:
def ls(conn, Bucket, Prefix=''):
objs = conn.list_objects(Bucket=Bucket, Prefix=Prefix)
yield [o['Key'] for o in objs.get('Contents', [])]
while objs.get('IsTruncated'):
objs = conn.list_objects(Bucket=Bucket, Prefix=Prefix,
Marker=objs['NextMarker'])
yield [o['Key'] for o in objs['Contents']]
# usage: iterate over batches of up to 1000 keys
for batch in ls(conn, 'testdir'):
print(batch)
Scientific data – numpy and zarr
s3fs.S3Map bridges S3 and array formats, enabling direct cloud storage of scientific datasets.
import numpy as np, zarr, s3fs fs = get_fs() # this function defined earlier in s3fs section store = s3fs.S3Map(root='testdir/mydata', s3=fs, check=False) # zarr — chunked, compressed, cloud-native array format data = np.ones((1000, 1000)) zarr.save(store, data) # write arr = zarr.load(store) # read zarr.open(store).info # inspect shape, chunks, compression
Zarr is particularly well-suited to S3 because its chunked design allows reading only the parts of an array you need — no need to download an entire file. For further reading: https://zarr.readthedocs.io/en/stable/
Notes
rcloneis another option for syncing data to/from S3 and is worth considering if you are already familiar with it. Configuration is similar to the above.- Bucket policies are not covered here. Policy support varies significantly between S3 implementations (AWS, Ceph, OpenStack Swift, MinIO, etc.) — features available on AWS may not be supported on this service. Contact your administrator before relying on policy-based access control.
----
Additional information: presentation notes and talk