S3 Object Storage
Simple Storage Service (S3)
The S3 service is easy to use, with a simple web services (REST) interface to store and retrieve any amount of data from anywhere on the web.
Each SWITCHengines region (LS or ZH) has a separate object storage S3 service:
Therefore, you can decide in which region you want S3 to store your data in. Both S3 services are accessible from everywhere, so you can use the S3 in the region LS with your SWITCHengines in the region ZH.
You must specify a region when you create your S3 bucket. Within that region, your objects are redundantly stored on multiple devices.
To use the SWITCHengines S3 service, you need your EC2 credentials. The EC2 credentials are composed of an Access Key and a Secret Key.
The EC2 credentials are valid per user in a given project (tenant) for both region (LS or ZH).
List your EC2 Credentials
- Make sure you have the OpenStack CLI tool installed (see https://docs.openstack.org/python-openstackclient/latest/)
- Source your OpenStack credentials
- Run the OpenStack command:
openstack ec2 credentials list
- You should get your EC2 Access Key and Secret Key for the project.
S3 Client and API Libraries
There are numerous tools to access and manage a S3 object storage. Here are some of the most used ones.
s3cmd Command Line Tool
The s3cmd is a command line tool and client for uploading, retrieving and managing data in S3 object storage service.
Configuration for s3cmd
By default s3cmd uses a configuration $HOME/.s3cfg file. You should define your EC2 credentials and service URL in this file:
use_https = True
access_key = asfqwe34sdtasg3345df2345f234fds3
secret_key = 342sdgf76sdfg56shg234fg45fdg435d
# region LS (comment out the following lines)
#host_base = os.unil.cloud.switch.ch
#host_bucket = %(bucket)s.os.unil.cloud.switch.ch
# region ZH (comment out the following lines)
host_base = os.zhdk.cloud.switch.ch
host_bucket = %(bucket)s.os.zhdk.cloud.switch.ch
The host_base and host_bucket parameters depend on the region (LS or ZH) you want to use.
Using the s3cmd, you can create a public bucket, and add public objects to it:
$ s3cmd mb s3://valery-public --acl-public
$ s3cmd put ghibli-ghost.png s3://valery-public/ --acl-public
$ s3cmd put clockwork-orange-big.png s3://valery-public/ --acl-public
Because the bucket and the objects are public, you can access them directly from:
Python boto library
The python boto library can also be used to access and manage a S3 object storage service.
See the An Introduction to boto’s S3 interface document for some examples.
Most tools and library that talk to S3 can be used with the SWITCHengines Object Store. Here is a list of tools that we know are working: