We are hosting our thumbor image resizing service on Heroku, with image store in aws s3. The Thumbor service access s3 via the aws plugin. Recently we observe a behavior on our thumbor service which I don't understand.
Use case: our client application was sending a resize request to resize an image that does not exist in aws S3.
expected behavior:
the thumbor service fetch image in s3.
s3 returns 404 not found
thumbor return 404 to heroku, the router returns 404 to client app.
what we observe: under certain situation(I cannot reproduce this consistently). s3 returns 404 but thumbor does not let the heroku router know. As a result the heroku wait for 30s and return the request as 503 time out.
The flowing are the logs
2016-05-31T19:38:15.094468+00:00 app[web.1]: 2016-05-31 19:38:15 thumbor:WARNING ERROR retrieving image from S3 [bucket]/3C6A3A84-F249-458B-9EFA-BF9BC863874B: {'ResponseMetadata': {'HTTPStatusCode': 404, 'RequestId': '3D61A8CBB187D846', 'HostId': 'C1qYC9Au42J0Salt1SVlCkcvcrKcQv4dltwOCdwGNF1TUFScWpkHb1qC++ZBJ0JzVqQlXW0xONU='}, 'Error': {'Key': '[bucket]/3C6A3A84-F249-458B-9EFA-BF9BC863874B', 'Code': 'NoSuchKey', 'Message': 'The specified key does not exist.'}}
2016-05-31T19:38:14.777549+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/bALh_vgGXd7e_J7kZ0GhyE_lhZ0=/150x150/[bucket]/3C6A3A84-F249-458B-9EFA-BF9BC863874B" host=heroku.app.com request_id=67d87ea3-8010-4fbe-8c29-b2b7298f1cbc fwd="54.162.233.176,54.240.144.43" dyno=web.1 connect=5ms service=30000ms status=503 bytes=0
I am wondering if anyone can help to understand why thumbor hangs?
Many thanks in advance!
This issue on the tc_aws library we are using. It wasn't executing callback functions when 404 is returned from S3. We were at version 2.0.10. After upgrade the library, the problem is fixed.