I am attempting to use the library s3fs in a tornado application. The easiest approach would be to make a small function that fetches some data from s3 (using s3fs) and wrap that in run_in_executor().
Is there any advantage to writing a custom library to fetch from s3 using tornado's AsyncHTTPClient instead?
(Note: I am aware of botoaio and others. For my particular use-case I require the output to be in the format provided by s3fs.)
AsyncHTTPClient is more efficient than running a synchronous library with run_in_executor. The difference is mainly the memory needed for thread stacks at high levels of concurrency. Performance will generally be similar as long as you have enough threads and memory for your level of concurrency. So unless you're doing a lot of s3 operations in parallel, there's little reason to invest a lot of work in converting everything to async interfaces.