I am using the AWS-SDK in Go to create a AWS config and upload/put files to the bucket. This is done using a AWS session.
Snippet:
s3Config := &aws.Config{
Credentials: credentials.NewStaticCredentials(a.AccessKey, a.SecretKey, a.AccessToken),
Endpoint: aws.String(s3URL),
Region: aws.String(region),
S3ForcePathStyle: true,
DisableSSL: true, }
newSession := session.NewSession(s3Config)
_, err = newSession.PutObject(&s3.PutObjectInput{
Bucket: aws.String(backupLocation.BucketName),
Key: aws.String(key),
})
I am trying to replicate the same with python boto3:
config = Config(
s3 = {
"addressing_style": "path",
"payload_signing_enabled": True,
"us_east_1_regional_endpoint": "regional"
}
)
s3_client = session.client('s3', region_name=region, endpoint_url=endpoint, verify=validate_ssl_certificates, config=config)
s3_client.head_bucket(Bucket=bucket_name)
s3_client.put_object(Bucket=bucket_name,Key=filename)
Both are working fine, but there is an error message when using the Go SDK that is not presented while running the same in Python.
If the bucket is of a region other than "us-east-1" for example "us-west-2" and if endpoint provided is "https://s3.amazonaws.com", on running PutObject() from the AWS-SDK in Go, i get the error:
S3 Error: BucketRegionError| incorrect region, the bucket is not in 'us-west-2' region at endpoint 'https://s3.amazonaws.com'
But it works with boto3, I guess it ignores the endpoint provided? I want to modify my code to fail with an error even with the boto3. So that the input can be corrected to: "https://s3.us-west-2.amazonaws.com"
My API was trying to check if the endpoint and region was correct according to the bucket region.
For example:
If the bucket "my-bucket" was created in region "us-west-2" but the endpoint input was given as "https://s3.amazonaws.com"
If you use Python's boto3 by trying to run a head_bucket() to validate the inputs it would successfully return a valid response.
However, this seems incorrect, because the endpoint url should be "https://s3.us-west-2.amazonaws.com" as "https://s3.amazonaws.com" defaults to "us-east-1"
Internally boto3 manipulates the endpoints on a failure response and retries by creating an endpoint url based on the region. So, the boto3 API calls completes successfully.
I solved this by making a REST call, to HeadBucket(), see: https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html If the region or endpoint is incorrect, the error code of 301 is returned.