...
check out their API for function signatures andalso their documentation for more examples.
Botocore - low level interface
- Botocore is a low-level interface to a growing number of Amazon Web Services.
- Botocore serves as the foundation for the AWS-CLI command line utilities.
- Sort-of oriented towards library builders
Save the credentials as follows (user is free to choose the convenient file name and file format)
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
service_name: s3
aws_access_key_id: HPC_user
aws_secret_access_key: t1H13sOUBD/H7NuL
endpoint_url: https://hssrv2.dmawi.de:635
region_name: bhv
verify: /Users/pasili001/Documents/HSM_S3gw.cert.pem |
Write a utility function to read the config file
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
import os
import yaml
import boto3
def get_connection():
with open(os.path.expanduser("~/.s3fs_boto")) as fid:
credentials = yaml.safe_load(fid)
return boto3.client(**credentials) |
Listing buckets and objects
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
>>> conn = get_connection()
>>> # Listing buckets
>>> print(conn.list_buckets())
{'Buckets': [{'CreationDate': datetime.datetime(2024, 4, 7, 15, 57, 46, 944296, tzinfo=tzoffset(None, 7200)),
'Name': 'testdir'}],
'Owner': {'DisplayName': '', 'ID': 'HPC_user'},
'ResponseMetadata': {'HTTPHeaders': {'connection': 'close',
'content-length': '315',
'content-type': 'application/xml',
'date': 'Sun, 07 Apr 2024 21:50:03 GMT',
'server': 'VERSITYGW'},
'HTTPStatusCode': 200,
'RetryAttempts': 0}}
>>>
>>> # filtering down the results just to show the bucket names
>>> for bucket in conn.list_buckets().get('Buckets'):
... print(bucket['Name'])
...
'testdir'
>>> # Listing objects
>>> objs = conn.list_objects(Bucket='testdir')
>>> print(obj)
{'Delimiter': '',
'EncodingType': '',
'IsTruncated': False,
'Marker': '',
'MaxKeys': 1000,
'Name': 'testdir',
'NextMarker': '',
'Prefix': '',
'ResponseMetadata': {'HTTPHeaders': {'connection': 'close',
'content-length': '67702',
'content-type': 'application/xml',
'date': 'Sun, 07 Apr 2024 21:58:15 GMT',
'server': 'VERSITYGW'},
'HTTPStatusCode': 200,
'RetryAttempts': 0},
'Contents': [{'ETag': '5f0137574247761b438aa508333f487d',
'Key': 'tmp.csv',
'LastModified': datetime.datetime(2024, 4, 6, 1, 11, 30, 890787, tzinfo=tzoffset(None, 7200)),
'Size': 385458,
'StorageClass': 'STANDARD'},
{'ETag': 'd776a1b6e8dc88615118832c552afd4c',
'Key': 'demo-airtemp/lon/0',
'LastModified': datetime.datetime(2024, 4, 7, 15, 58, 49, 37104, tzinfo=tzoffset(None, 7200)),
'Size': 118,
'StorageClass': 'STANDARD'},
{'ETag': 'ffe3e35a2a10544db446cb5ffb64516b',
'Key': 'demo-airtemp/time/.zarray',
'LastModified': datetime.datetime(2024, 4, 7, 15, 58, 49, 410103, tzinfo=tzoffset(None, 7200)),
'Size': 319,
'StorageClass': 'STANDARD'},
{'ETag': 'c3469e3ac4f2746bdb750335dbcd104a',
'Key': 'demo-airtemp/time/.zattrs',
'LastModified': datetime.datetime(2024, 4, 7, 15, 58, 49, 520103, tzinfo=tzoffset(None, 7200)),
'Size': 172,
'StorageClass': 'STANDARD'},
...
...
{'ETag': '7c6e83fce9aa546ec903ca93f036a2fd',
'Key': 'demo-airtemp/time/0',
'LastModified': datetime.datetime(2024, 4, 7, 15, 58, 49, 630102, tzinfo=tzoffset(None, 7200)),
'Size': 2549,
'StorageClass': 'STANDARD'}]}
|
The output for listing the objects is truncated on purpose to avoid filling up this page. Unlike the other clients, botocore provides a lot of metadata information related to buckets and objects.