amazon-s3

AWS Prefix Hierarchy vs Request Rate Limit: Does request to A/B/C count against A/B and A/?

There is something that is quite unclear to me about aws s3 prefixes, which i would hope someone could help me with.

I am wondering if when i have the prefixes

A/B/C/ and A/B/D/

Any request against A/B/C/ or A/B/D/ count against A/B/ and ultimately A/ in term of request rate ?

Hence that the parent prefix limit apply to the child prefix.

Solution

  • Please refer this similar question for more details S3 - What Exactly Is A Prefix? And what Ratelimits apply?

    Based on response from AWS support as below,

    Earlier S3 supported 100 PUT/LIST/DELETE requests per second and 300 GET requests per second. To achieve higher performance, a random hash / prefix schema had to be implemented. Since last year the request rate limits increased to 3,500 PUT/POST/DELETE and 5,500 GET requests per second. This increase is often enough for applications to mitigate 503 SlowDown errors without having to randomize prefixes.

    However, if the new limits are not sufficient, prefixes would need to be used. A prefix has no fixed number of characters. It is any string between a bucket name and an object name, for example:

    • bucket/folder1/sub1/file
    • bucket/folder1/sub2/file
    • bucket/1/file
    • bucket/2/file

    Prefixes of the object 'file' would be: /folder1/sub1/ , /folder1/sub2/, /1/, /2/. In this example, if you spread reads across all four prefixes evenly, you can achieve 22,000 requests per second.

    It means each of your prefixes A/B/C/ and A/B/D/ has separate limit, and these prefixes are not adding limit towards root prefixes A/B/ and A/

    Also refer: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html for Organizing objects using prefixes