amazon-web-servicesamazon-s3boto3amazon-iamaws-sts

boto3 s3 copy HeadObject 403 Forbidden


Source and destination buckets are in the same region but different AWS accounts. The copy code that fails:

import boto3
from datetime import datetime
assumed_role = boto3.client('sts').assume_role(
  RoleArn='arn:aws:iam::DESTACCT:role/copy-role',
  RoleSessionName='RoleSessionName' + datetime.now().strftime('%Y%M%d%H%m%s'))
args = {
  'aws_access_key_id': assumed_role['Credentials']['AccessKeyId'],
  'aws_secret_access_key': assumed_role['Credentials']['SecretAccessKey'],
  'aws_session_token': assumed_role['Credentials']['SessionToken']}
session = boto3.Session(**args)
dest_bucket = session.resource('s3').Bucket('dest-bucket')
copy_source = {'Key': 'source-prefix/someobject', 'Bucket': 'source-bucket'}
dest_bucket.copy(CopySource=copy_source, Key='dest-prefix/someobject')
[...backtrace...]
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Permissions on copy-role:

{
      "Action": "s3:Put*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::dest-bucket/dest-prefix/*"
}

{
      "Action": ["s3:List*","s3:Get*"],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::source-bucket/source-prefix/*",
        "arn:aws:s3:::source-bucket/source-prefix"
      ]
    }

source-bucket includes these clauses in its bucket policy:

{
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::DESTACCT:role/copy-role"
      },
      "Action": "s3:List*",
      "Resource": "arn:aws:s3:::source-bucket"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::DESTACCT:role/copy-role"
      },
      "Action": [
        "s3:List*",
        "s3:Get*"
      ],
      "Resource": [
        "arn:aws:s3:::source-bucket/source-prefix/*",
        "arn:aws:s3:::source-bucket/source-prefix"
      ]
    }

I note the following from the copy() documentation:

SourceClient (botocore or boto3 Client) – The client to be used for operation that may happen at the source object. For example, this client is used for the head_object that determines the size of the copy. If no client is provided, the current client is used as the client for the source object.

However, it seems like that should not be necessary since copy-role is allowed s3:Get* and s3:List* on source-bucket/source-prefix.

I logged all HTTP traffic and confirmed that the HEAD operation that's failing is against the source bucket:

HEAD /source-prefix/someobject HTTP/1.1
Host: source-bucket.s3.amazonaws.com

Solution

  • I suspect that the problem lies in this part of the Bucket Policy on the Source bucket:

        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::DESTACCT:role/copy-role"
          },
          "Action": [
            "s3:List*",
            "s3:Get*"
          ],
          "Resource": [
            "arn:aws:s3:::source-bucket/source-prefix/*",
            "arn:aws:s3:::source-bucket/source-prefix"
          ]
        }
    

    It is granting ListBucket permissions, but ListBucket requires permissions on the bucket itself rather than a Prefix within the bucket.

    Using the example from Bucket policy examples - Amazon Simple Storage Service, you could change it to:

        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::DESTACCT:role/copy-role"
          },
          "Action": "s3:s3:ListBucket",
          "Resource": "arn:aws:s3:::source-bucket",
          "Condition": {
              "StringLike": {
                  "s3:prefix": ["source-prefix/*"]
              }
          }
        }
    

    You will likely need to do similar in the IAM Role.