S3 Object Storage

SWITCHengines offers an object storage service. The service is compatible with the Amazon S3 API, but some AWS features are not supported.

Simple Storage Service (S3)

The S3 service is easy to use, with a simple web services (REST) interface to store and retrieve any amount of data from anywhere on the web.

It is based on Ceph Object Gateway, and the S3 API have the following set of features (compared to Amazon S3).

S3 Region

Each SWITCHengines region (LS or ZH) has a separate object storage S3 service:

Therefore, you can decide in which region you want S3 to store your data in. Both S3 services are accessible from everywhere, so you can use the S3 in the region LS with your SWITCHengines in the region ZH.

You must specify a region when you create your S3 bucket. Within that region, your objects are redundantly stored on multiple devices.

EC2 Credentials

To use the SWITCHengines S3 service, you need your EC2 credentials. The EC2 credentials are composed of an Access Key and a Secret Key.

The EC2 credentials are valid per user in a given project (tenant) for both region (LS or ZH).

List your EC2 Credentials
  1. Make sure you have the OpenStack CLI tool installed (see https://docs.openstack.org/python-openstackclient/latest/)
  2. Source your OpenStack credentials
  3. Run the OpenStack command:
    openstack ec2 credentials list
  4. You should get your EC2 Access Key and Secret Key for the project.

S3 Client and API Libraries

There are numerous tools to access and manage a S3 object storage. Here are some of the most used ones.

s3cmd Command Line Tool

The s3cmd is a command line tool and client for uploading, retrieving and managing data in S3 object storage service.

Configuration for s3cmd

By default s3cmd uses a configuration $HOME/.s3cfg file. You should define your EC2 credentials and service URL in this file:

[default]
use_https = True
access_key = asfqwe34sdtasg3345df2345f234fds3
secret_key = 342sdgf76sdfg56shg234fg45fdg435d
# region LS (comment out the following lines)
#host_base = os.unil.cloud.switch.ch
#host_bucket = %(bucket)s.os.unil.cloud.switch.ch
# region ZH (comment out the following lines)
host_base = os.zhdk.cloud.switch.ch
host_bucket = %(bucket)s.os.zhdk.cloud.switch.ch

The host_base and host_bucket parameters depend on the region (LS or ZH) you want to use.

Example

Using the s3cmd, you can create a public bucket, and add public objects to it:

$ s3cmd mb s3://valery-public --acl-public

$ s3cmd put ghibli-ghost.png s3://valery-public/ --acl-public
$ s3cmd put clockwork-orange-big.png s3://valery-public/ --acl-public

Because the bucket and the objects are public, you can access them directly from:

https://valery-public.os.zhdk.cloud.switch.ch/clockwork-orange-big.png

https://valery-public.os.zhdk.cloud.switch.ch/ghibli-ghost.png

Python boto library

The python boto library can also be used to access and manage a S3 object storage service.

See the An Introduction to boto’s S3 interface document for some examples.

Other Tools

Most tools and library that talk to S3 can be used with the SWITCHengines Object Store. Here is a list of tools that we know are working: