amazon-web-servicesamazon-s3boto3aws-sdk-gobotocore

AWS S3 validate endpoint with region


I am using the AWS-SDK in Go to create a AWS config and upload/put files to the bucket. This is done using a AWS session.

Snippet:

    s3Config := &aws.Config{
        Credentials:      credentials.NewStaticCredentials(a.AccessKey, a.SecretKey, a.AccessToken),
        Endpoint:         aws.String(s3URL),
        Region:           aws.String(region),
        S3ForcePathStyle: true,
        DisableSSL:       true, }
    newSession := session.NewSession(s3Config)
    _, err = newSession.PutObject(&s3.PutObjectInput{
        Bucket: aws.String(backupLocation.BucketName),
        Key:    aws.String(key),
    })

I am trying to replicate the same with python boto3:

    config = Config(
        s3 = {
            "addressing_style": "path",
            "payload_signing_enabled": True,
            "us_east_1_regional_endpoint": "regional"
        }
    )
    s3_client = session.client('s3', region_name=region, endpoint_url=endpoint, verify=validate_ssl_certificates, config=config)
    s3_client.head_bucket(Bucket=bucket_name)
    s3_client.put_object(Bucket=bucket_name,Key=filename)

Both are working fine, but there is an error message when using the Go SDK that is not presented while running the same in Python.

If the bucket is of a region other than "us-east-1" for example "us-west-2" and if endpoint provided is "https://s3.amazonaws.com", on running PutObject() from the AWS-SDK in Go, i get the error:

S3 Error: BucketRegionError| incorrect region, the bucket is not in 'us-west-2' region at endpoint 'https://s3.amazonaws.com'

But it works with boto3, I guess it ignores the endpoint provided? I want to modify my code to fail with an error even with the boto3. So that the input can be corrected to: "https://s3.us-west-2.amazonaws.com"


Solution

  • My API was trying to check if the endpoint and region was correct according to the bucket region.
    For example:
    If the bucket "my-bucket" was created in region "us-west-2" but the endpoint input was given as "https://s3.amazonaws.com"
    If you use Python's boto3 by trying to run a head_bucket() to validate the inputs it would successfully return a valid response.

    However, this seems incorrect, because the endpoint url should be "https://s3.us-west-2.amazonaws.com" as "https://s3.amazonaws.com" defaults to "us-east-1"

    Internally boto3 manipulates the endpoints on a failure response and retries by creating an endpoint url based on the region. So, the boto3 API calls completes successfully.

    I solved this by making a REST call, to HeadBucket(), see: https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html If the region or endpoint is incorrect, the error code of 301 is returned.