S3 Access Policy

Since the upgrade to Ceph "Luminous" in February 2018, it is possible to use S3 bucket policy instead of the S3 bucket/object ACL. 

Using S3 bucket policy is more light-weight, configurable, and scalable than the S3 ACL. In particular if the bucket contains a lot of objects, updating the ACL does not scale, and will take forever.

It is not recommended to use S3 access control list (ACL) anymore. Managing S3 bucket and object ACLs is cumbersome, limited, and does not scale.

S3 Bucket Policy

S3 bucket policy allows you to grant access to your bucket to other projects. Access policies are more fine-grained, more scalable, and allow a much better control of your data.

References

Limitations

  • Because of the Ceph implementation and OpenStack Keystone integration, it is only possible to grant access right at the project level.
  • It is NOT possible to grant access right directly to a user, but only to the project the user is member of.
  • Only S3 bucket policy is available, S3 user policy is not implemented in Ceph S3.
  • Projects must be identified with their project ID, not project name.

Principal Identifier

In a S3 access policy file, the project accessing your bucket must be identified using a Principal identifier. See https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-bucket-user-policy-specifying-principal-intro.html

You must use the OpenStack project ID, not the name, to identify the principal you want to grant access to your bucket.

Use an AWS account ID (ARN) format to identify the project accessing your bucket:

"Principal": {
"AWS": "arn:aws:iam::PROJECT_ID:root"
}

or to identify multiple projects:

"Principal": {
"AWS": [
"arn:aws:iam::PROJECT_ID_1:root",
"arn:aws:iam::PROJECT_ID_2:root"
]
}

Policy: Read Only Bucket and Objects

If the project owner (e.g. OWNER_PROJECT) of the bucket (e.g. SHARED_BUCKET) want to share it read only with another project (e.g. READ_ONLY_PROJECT), the following policy can be used:

{
"Version": "2012-10-17",
"Id": "read-only",
"Statement": [
{
"Sid": "project-read",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::READ_ONLY_PROJECT_ID:root"
},
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
  • READ_ONLY_PROJECT_ID is the project ID of the project with read only access to the bucket.
  • the element "Version": "2012-10-17" is mandatory.

Then the bucket's owner uses the s3cmd command to apply and check the policy on the bucket:

s3cmd -c s3cfg-OWNER_PROJECT setpolicy read-only.json s3://SHARED_BUCKET
s3cmd -c s3cfg-OWNER_PROJECT info s3://SHARED_BUCKET

Policy: Read Only Bucket Restricted to "sub-directory" Objects

If the project owner (e.g. OWNER_PROJECT) of the bucket (e.g. SHARED_BUCKET) want to share read only a sub-directory structure with another project (e.g. READ_ONLY_PROJECT), the following policy can be used:

{
  "Version": "2012-10-17",
  "Id": "read-only-subdirectory",
  "Statement": [
    {
      "Sid": "project-list_bucket",
      "Effect": "Allow",
      "Principal": {
       "AWS": "arn:aws:iam::READ_ONLY_PROJECT_ID:root"
      },
      "Action": [
       "s3:ListBucket"
      ],
      "Resource": [
       "arn:aws:s3:::*"
      ]
    },
    {
     "Sid": "project-read_subdirectory",
     "Effect": "Allow",
     "Principal": {
      "AWS": "arn:aws:iam::READ_ONLY_PROJECT_ID:root"
},
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::SHARED_BUCKET/foo/bar/*"
]
}
]
}
  • the project-list_bucket statement is not mandatory, but without it the read only project can not list the bucket's objects, and therefore have to know the object path in advance.
  • READ_ONLY_PROJECT_ID is the project ID of the project with read only access to the bucket.
  • SHARED_BUCKET/foo/bar/* is the bucket/objects sub-directory prefix to restrict access to.
  • the element "Version": "2012-10-17" is mandatory. 

Then the bucket's owner uses the s3cmd command to apply and check the policy on the bucket:

s3cmd -c s3cfg-OWNER_PROJECT setpolicy read-only-subdirectory.json s3://SHARED_BUCKET
s3cmd -c s3cfg-OWNER_PROJECT info s3://SHARED_BUCKET

Policy: Read and Write Bucket and Objects

If the project owner (e.g. OWNER_PROJECT) of the bucket (e.g. SHARED_BUCKET) want to share it read and write with another project (e.g. READ_ONLY_PROJECT), the following policy can be used:

{
"Version": "2012-10-17",
"Id": "read-write",
"Statement": [
{
"Sid": "project-read_write",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::OWNER_PROJECT_ID:root",
"arn:aws:iam::READ_WRITE_PROJECT_ID:root"
]
},
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
  • OWNER_PROJECT_ID is the project ID of the project sharing the bucket. The bucket owner must be listed explicitly in the Principal AWS list.
  • READ_WRITE_PROJECT_ID is the project ID of the project with read and write access to the bucket.
  • the element "Version": "2012-10-17" is mandatory.

Then the bucket's owner uses the s3cmd command to apply and check the policy on the bucket:

s3cmd -c s3cfg-OWNER_PROJECT setpolicy read-write.json s3://SHARED_BUCKET
s3cmd -c s3cfg-OWNER_PROJECT info s3://SHARED_BUCKET