One place for hosting & domains

      Bucket

      How to Enact Access Control Lists (ACLs) and Bucket Policies with Linode Object Storage


      Updated by Linode

      Contributed by
      Linode

      Linode Object Storage allows users to share access to objects and buckets with other Object Storage users. There are two mechanisms for setting up sharing: Access Control Lists (ACLs), and bucket policies. These mechanisms perform similar functions: both can be used to restrict and grant access to Object Storage resources.

      In this guide you will learn:

      Before You Begin

      • This guide will use the s3cmd command line utility to interact with Object Storage. For s3cmd installation and configuration instructions, visit our How to Use Object Storage guide.

      • You’ll also need the canonical ID of every user you wish to grant additional permissions to.

      Retrieve a User’s Canonical ID

      Follow these steps to determine the canonical ID of the Object Storage users you want to share with:

      1. The following command will return the canonical ID of a user, given any of the user’s buckets:

        s3cmd info s3://other-users-bucket
        

        Note

        The bucket referred to in this section is an arbitrary bucket on the target user’s account. It is not related to the bucket on your account that you would like to set ACLs or bucket policies on.

        There are two options for running this command:

        • The users you’re granting or restricting access to can run this command on one of their buckets and share their canonical ID with you, or:

        • You can run this command yourself if you have use of their access tokens (you will need to configure s3cmd to use their access tokens instead of your own).

      2. Run the above command, replacing other-users-bucket with the name of the bucket. You’ll see output similar to the following:

          
        s3://other-users-bucket/ (bucket):
        Location:  default
        Payer:     BucketOwner
        Expiration Rule: none
        Policy:    none
        CORS:      none
        ACL:       a0000000-000a-0000-0000-00d0ff0f0000: FULL_CONTROL
        
        
      3. The canonical ID of the owner of the bucket is the long string of letters, dashes, and numbers found in the line labeled ACL, which in this case is a0000000-000a-0000-0000-00d0ff0f0000.

      4. Alternatively, you may be able to retrieve the canonical ID by curling a bucket and retrieving the Owner ID field from the returned XML. This method is an option when both of these conditions are true:

        • The bucket has objects within it and has already been set to public (with a command like s3cmd setacl s3://other-users-bucket --acl-public).
        • The bucket has not been set to serve static websites.
      5. Run the curl command, replacing the bucket name and cluster URL with the relevant values:

        curl other-users-bucket.us-east-1.linodeobjects.com
        
      6. This will result in the following output:

        <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
            <Name>acl-bucket-example</Name>
            <Prefix/>
            <Marker/>
            <MaxKeys>1000</MaxKeys>
            <IsTruncated>false</IsTruncated>
            <Contents>
            <Key>cpanel_one-click.gif</Key>
            <LastModified>2019-11-20T16:52:49.946Z</LastModified>
            <ETag>"9aeafcb192a8e540e7be5b51f7249e2e"</ETag>
            <Size>961023</Size>
            <StorageClass>STANDARD</StorageClass>
            <Owner>
                <ID>a0000000-000a-0000-0000-00d0ff0f0000</ID>
                <DisplayName>a0000000-000a-0000-0000-00d0ff0f0000</DisplayName>
            </Owner>
            <Type>Normal</Type>
            </Contents>
        </ListBucketResult>
        

        In the above output, the canonical ID is a0000000-000a-0000-0000-00d0ff0f0000.

      ACLs vs Bucket Policies

      ACLs and bucket policies perform similar functions: both can restrict or grant access to buckets. ACLs can also restrict or grant access to individual objects, but they don’t offer as many fine-grained access modes as bucket policies.

      How to Choose Between ACLs and Bucket Policies

      If you can organize objects with similar permission needs into their own buckets, then it’s strongly suggested that you use bucket policies. However, if you cannot organize your objects in this fashion, ACLs are still a good option.

      ACLs offer permissions with less fine-grained control than the permissions available through bucket policies. If you are looking for more granular permissions beyond read and write access, choose bucket policies over ACLs.

      Additionally, bucket policies are created by applying a written bucket policy file to the bucket. This file cannot exceed 20KB in size. If you have a policy with a lengthy list of policy rules, you may want to look into ACLs instead.

      Note

      ACLs and bucket policies can be used at the same time. When this happens, any rule that limits access to an Object Storage resource will override a rule that grants access. For instance, if an ACL allows a user access to a bucket, but a bucket policy denies that user access, the user will not be able to access that bucket.

      ACLs

      Access Control Lists (ACLs) are a legacy method of defining access to Object Storage resources. You can apply an ACL to a bucket or to a specific object. There are two generalized modes of access: setting buckets and/or objects to be private or public. A few other more granular settings are also available.

      With s3cmd, you can set a bucket to be public with the setacl command and the --acl-public flag:

      s3cmd setacl s3://acl-example --acl-public
      

      This will cause the bucket and its contents to be downloadable over the general Internet.

      To set an object or bucket to private, you can use the setacl command and the --acl-private flag:

      s3cmd setacl s3://acl-example --acl-private
      

      This will prevent users from accessing the bucket’ contents over the general Internet.

      Other ACL Permissions

      The more granular permissions are:

      Permission Description
      read Users with can list objects within a bucket
      write Users can upload objects to a bucket and delete objects from a bucket.
      read_acp Users can read the ACL currently applied to a bucket.
      write_acp Users can change the ACL applied to the bucket.
      full_control Users have read and write access over both objects and ACLs.
      • Setting a permission: To apply these more granular permissions for a specific user with s3cmd, use the following setacl command with the --acl-grant flag:

        s3cmd setacl s3://acl-example --acl-grant=PERMISSION:CANONICAL_ID
        

        Substitute acl-example with the name of the bucket (and the object, if necessary), PERMISSION with a permission from the above table, and CANONICAL_ID with the canonical ID of the user to which you would like to grant permissions.

      • Revoking a permission: To revoke a specific permission, you can use the setacl command with the acl-revoke flag:

        s3cmd setacl s3://acl-example --acl-revoke=PERMISSION:CANONICAL_ID
        

        Substitute the bucket name (and optional object), PERMISSION, and CANONICAL_ID with your relevant values.

      • View current ACLs: To view the current ACLs applied to a bucket or object, use the info command, replacing acl-example with the name of your bucket (and object, if necessary):

        s3cmd info s3://acl-example
        

        You should see output like the following:

          
        s3://acl-bucket-example/ (bucket):
           Location:  default
           Payer:     BucketOwner
           Expiration Rule: none
           Policy:    none
           CORS:      b'<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><CORSRule><AllowedMethod>GET</AllowedMethod><AllowedMethod>PUT</AllowedMethod><AllowedMethod>DELETE</AllowedMethod><AllowedMethod>HEAD</AllowedMethod><AllowedMethod>POST</AllowedMethod><AllowedOrigin>*</AllowedOrigin><AllowedHeader>*</AllowedHeader></CORSRule></CORSConfiguration>'
           ACL:       *anon*: READ
           ACL:       a0000000-000a-0000-0000-00d0ff0f0000: FULL_CONTROL
           URL:       http://us-east-1.linodeobjects.com/acl-example/
        
        

        Note

        The owner of the bucket will always have the full_control permission.

      Bucket Policies

      Bucket policies can offer finer control over the types of permissions you can grant to a user. Below is an example bucket policy written in JSON:

      bucket_policy_example.txt
       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      
      {
        "Version": "2012-10-17",
        "Statement": [{
          "Effect": "Allow",
          "Principal": {
            "AWS": [
              "arn:aws:iam:::a0000000-000a-0000-0000-00d0ff0f0000"
            ]
          },
          "Action": [
            "s3:PutObject",
            "s3:GetObject",
            "s3:ListBucket"
          ],
          "Resource": [
            "arn:aws:s3:::bucket-policy-example/*"
          ]
        }]
      }

      This policy allows the user with the canonical ID a0000000-000a-0000-0000-00d0ff0f0000, known here as the “principal”, to interact with the bucket, known as the “resource”. The “resource” that is listed (bucket-policy-example) is the only bucket the user will have access to.

      Note

      The principal (a.k.a. the user) must have the prefix of arn:aws:iam:::, and the resource (a.k.a. the bucket) must have the prefix of arn:aws:s3:::.

      The permissions are specified in the Action array. For the current example, these are:

      The Action and Principal.AWS fields of the bucket policy are arrays, so you can easily add additional users and permissions to the bucket policy, separating them by a comma. To grant permissions to all users, you can supply a wildcard (*) to the Principal.AWS field.

      If you instead wanted to deny access to the user, you could change the Effect field to Deny.

      Enable a Bucket Policy

      To enable the bucket policy, use the setpolicy s3cmd command, supplying the file name of the bucket policy as the first argument, and the S3 bucket address as the second argument:

      s3cmd setpolicy bucket_policy_example.txt s3://bucket-policy-example
      

      To ensure that it has been applied correctly, you can use the info command:

      s3cmd info s3://bucket-policy-example
      

      You should see output like the following:

        
      s3://bucket-policy-example/ (bucket):
         Location:  default
         Payer:     BucketOwner
         Expiration Rule: none
         Policy:    b'{n  "Version": "2012-10-17",n  "Statement": [{n    "Effect": "Allow",n    "Principal": {"AWS": ["arn:aws:iam:::a0000000-000a-0000-0000-00d0ff0f0000"]},n    "Action": ["s3:PutObject","s3:GetObject","s3:ListBucket"],n    "Resource": [n      "arn:aws:s3:::bucket-policy-example/*"n    ]n  }]n}'
         CORS:      none
         ACL:       a0000000-000a-0000-0000-00d0ff0f0000: FULL_CONTROL
      
      

      Note

      The policy is visible in the output.

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link

      Bucket Versioning with Linode Object Storage


      Updated by Linode

      Contributed by
      Linode

      Note

      Linode Object Storage is currently in a closed early access Beta, and you may not have access to Object Storage through the Cloud Manager or other tools. To request access to the Early Access Program (EAP), open up a Customer Support ticket or e-mail [email protected] Beta access is completely free.

      Because Object Storage is in Beta, there may be breaking changes to how you access and manage Object Storage. This guide will be updated to reflect these changes if and when they occur. To view the changes that happen to this guide over time, visit the source text for it on GitHub.

      Linode Object Storage allows for bucket versioning so that you can retain different versions of your objects within buckets. This makes it easy to save older versions of objects, as well as quickly revert to an object’s previous state.

      In this guide, you will learn about:

      Before You Begin

      You should familiarize yourself with the basics of Linode Object Storage by reading the How to Use Linode Object Storage guide.

      You’ll also need to have a bucket with which you want to enable bucket versioning. Follow the Create a Bucket section of the How to Use Linode Object Storage guide if you do not already have one.

      Bucket Versioning

      Bucket versioning works by saving separate versions of objects in a bucket. When bucket versioning is enabled, an object that is uploaded to a bucket won’t overwrite a previous copy of that object. Instead, each version of the object is given a different ID number and saved.

      When attempting to view the object over HTTP or to download the object without supplying a version ID, the latest version of the object is returned. If you delete an object with bucket versioning enabled, a delete marker is inserted into the bucket to report that the object has been deleted, but the bucket will retain all previous versions of the object. Any further retrieval operations will return the most recent version of an object.

      Caution

      Every version of an object counts towards the monthly billable storage quota. While saving a few revisions is probably not something to worry about, large version controlled buckets with many thousands of objects will see a noticeable increase in storage space demands, and should be monitored carefully.

      Cyberduck

      Cyberduck is a GUI application that supports a wide variety of file transfer protocols, including S3-compatible Object Storage. For basic Cyberduck functionality, such as creating buckets and uploading objects, consult our How to Use Linode Object Storage guide.

      Note

      To use bucket versioning with Cyberduck:

      1. Enable bucket versioning for your bucket. Right-click or control-click on the bucket and select Info from the bucket’s context menu, or select File > Info from the menu bar.

        Select "Info" from the bucket's context menu.

      2. A settings menu will appear. Select the S3 menu heading.

        The "Info" settings menu.

      3. In the Versioning section, check the checkbox labeled Bucket Versioning.

        Select the S3 menu heading to view the S3 specific bucket settings.

      4. Your bucket will now retain previous versions of the objects within it. To test this functionality, create an example text document on your computer and add some text to it:

        test.txt
        1
        
        This is version 1 of the object.

        Upload this file to your bucket by dragging it into Cyberduck, or by selecting File > Upload from the menu bar.

      5. Now, edit the same file on your computer and make a small change. We will use this change to demonstrate bucket versioning.

        test.txt
        1
        
        This is version 2 of the object.

        Save the change and upload the file to your bucket just as you did in the previous step. Cyberduck will prompt you this time to confirm that you’d like to upload the file. Though the dropdown menu says Overwrite, the file will not be overwritten.

        Confirm that you'd like to upload the file to your bucket.

      6. You now have two objects in your bucket, though initially you may only see one. To view the different saved versions of your object, select View from the menu bar and click on Show Hidden Files.

        You should now see two files in your bucket, with the hidden file that you’ve just revealed being grayed-out. This grayed-out file is the older version of the file that you uploaded first.

        Note

        You may have to click the Refresh button in Cyberduck’s toolbar to see the hidden files.

        Viewing the hidden files, there are now two files in the bucket.

      7. Double click the grayed-out version of the file (the one with the earlier modified date) to download the file. Once opened, you’ll see that the file contains the contents of the first revision.

      8. To revert to a previous file revision, right-click or control-click on an object and select Revert. This will create a new object in the bucket, preserving the state of the previous two objects. At this point the current object will contain the contents of revision one, the second object will contain the contents of revision two, and the third object will contain to the contents of revision one. You can see, then, how bucket versioning works to maintain a stateful history.

        Note

        Each file has its own permissions. If you’d like to view objects via HTTP, then you’ll need to manually set the permissions for each object to Everyone each time you upload or revert to a different version of the object, as described in the How to Use Linode Object Storage guide.

      AWS CLI

      Because Linode Object Storage is compatible with S3, several tools that are designed to work with S3 can be adapted to Linode Object Storage, like the AWS CLI.



      Installing the AWS CLI

      The AWS CLI can be installed using pip, the Python package manager, on macOS and Linux:

      pip install awscli
      

      If you are using Windows, you can download the CLI by visiting the AWS CLI homepage.

      You’ll need to configure the AWS CLI before you can use it. To do so, run the configure command:

      aws configure
      

      You’ll be prompted to enter your AWS access key and AWS secret access key, but you can enter your Linode access key and Linode secret access key here instead. You can leave the other fields blank.

      To use bucket versioning with the AWS CLI:

      1. Turn on bucket versioning with the put-bucket-versioning command. Supply the Linode Object Storage endpoint and bucket name to the this command. In the below example, the bucket is named bucket-versioning-example:

        aws s3api put-bucket-versioning --endpoint=http://us-east-1.linodeobjects.com --bucket=bucket-versioning-example --versioning-configuration Status=Enabled
        
      2. The above command won’t prompt any response. To check on the status of bucket-versioning, issue the get-bucket-versioning command, again supplying the Linode Object Storage endpoint and your bucket name:

        aws s3api get-bucket-versioning --endpoint=http://us-east-1.linodeobjects.com --bucket=bucket-versioning-example
        

        You should see output like the following:

          
        {
            "Status": "Enabled",
            "MFADelete": "Disabled"
        }
        
        

        Here you can see that bucket versioning has been successfully applied to the bucket because the Status is set to Enabled.

      3. To test bucket versioning, create an example text document on your computer and add some text to it:

        test.txt
        1
        
        This is version 1 of the file.
      4. Upload the file to your bucket using the put-object command, again supplying the Linode Object Storage endpoint and your bucket name:

        aws s3api put-object --endpoint=http://us-east-1.linodeobjects.com  --bucket=bucket-versioning-example --key test.txt --body test.txt
        

        In the above command, the key is the name you would like to give the object after it is uploaded, and the body is the local filename of the object. These can be different.

        Note

        When retrieving the file you will need to give the object’s key rather than it’s local filename.

      5. You should see output like the following:

          
        {
            "ETag": ""9a7b64c98b066602b21f869ae7cd673a"",
            "VersionId": "4ACJQUNQ1ORV2oZ1SRE7l2M7nTjD1pv"
        }
        
        

        Take note of the VersionId, which will be referenced later.

      6. Now, modify the local file to indicate that this file is a different version than the one you previously uploaded:

        test.txt
        1
        
        This is version 2 of the file.
      7. Upload the modified version of the file to Object Storage:

        aws s3api put-object --endpoint=http://us-east-1.linodeobjects.com  --bucket=bucket-versioning-example --key test.txt --body test.txt
        

        You should see output like the following:

          
        {
            "ETag": ""38b0d2ff1c03df82aea67222983d337e"",
            "VersionId": "cdw7Lz4CPdm3wAJwBIJ6NhOUDqZo-Y7"
        }
        
        

        Note that the VersionId has changed to reflect the newer version of the object.

      8. You now have two versions of the object in your bucket. To view these objects, use the list-object-versions command:

        aws s3api list-object-versions --endpoint=http://us-east-1.linodeobjects.com --bucket=bucket-versioning-example
        

        You should see output like the following:

          
        {
            "Versions": [
                {
                    "ETag": ""38b0d2ff1c03df82aea67222983d337e"",
                    "Size": 6,
                    "StorageClass": "STANDARD",
                    "Key": "test.txt",
                    "VersionId": "cdw7Lz4CPdm3wAJwBIJ6NhOUDqZo-Y7",
                    "IsLatest": true,
                    "LastModified": "2019-09-18T11:51:33.070Z",
                    "Owner": {
                        "DisplayName": "a9354920-007a-4480-9271-06d3ff7f8426",
                        "ID": "a9354920-007a-4480-9271-06d3ff7f8426"
                    }
                },
                {
                    "ETag": ""9a7b64c98b066602b21f869ae7cd673a"",
                    "Size": 6,
                    "StorageClass": "STANDARD",
                    "Key": "test.txt",
                    "VersionId": "4ACJQUNQ1ORV2oZ1SRE7l2M7nTjD1pv",
                    "IsLatest": false,
                    "LastModified": "2019-09-18T11:51:23.041Z",
                    "Owner": {
                        "DisplayName": "a9354920-007a-4480-9271-06d3ff7f8426",
                        "ID": "a9354920-007a-4480-9271-06d3ff7f8426"
                    }
                }
            ]
        }
        
        

        Each version of the test.txt object is represented in the output.

      9. To download a previous version of an object, use the get-object command and supply the version-id of the earliest copy of the object:

        aws s3api get-object --endpoint=http://us-east-1.linodeobjects.com --bucket=bucket-versioning-example --key=test.txt --version-id=4ACJQUNQ1ORV2oZ1SRE7l2M7nTjD1pv test_old.txt
        

        Be sure to include an output filename for the downloaded object as the last parameter. In the above command the output file is named test_old.txt.

        You should see output like the following:

          
        {
            "AcceptRanges": "bytes",
            "LastModified": "Wed, 18 Sep 2019 11:51:23 GMT",
            "ContentLength": 6,
            "ETag": ""9a7b64c98b066602b21f869ae7cd673a"",
            "VersionId": "4ACJQUNQ1ORV2oZ1SRE7l2M7nTjD1pv",
            "ContentType": "binary/octet-stream",
            "Metadata": {
                "version-id": "r71GBm28-B2ynT89nAdAEIpXXZwUtqr"
            }
        }
        
        

        Open the downloaded file and confirm that the first version of the file was downloaded properly.

      10. To restore to a previous version of an object using the AWS CLI, you can download the version you want to restore to and then re-upload that file. Alternatively, if there is only one newer version of the object, you can delete the newer version of the object and Object Storage will automatically serve the next most recent version.

      More Information

      You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

      Find answers, ask questions, and help others.

      This guide is published under a CC BY-ND 4.0 license.



      Source link