[SOLVED] Profile argument in python s3fs

Profile argument in python s3fs

I'm trying to use s3fs in python to connect to an s3 bucket. The associated credentials are saved in a profile called 'pete' in ~/.aws/credentials:

[default]
aws_access_key_id=****
aws_secret_access_key=****

[pete]
aws_access_key_id=****
aws_secret_access_key=****

This seems to work in AWS CLI (on Windows):

$>aws s3 ls s3://my-bucket/ --profile pete
                       PRE other-test-folder/
                       PRE test-folder/

But I get a permission denied error when I use what should be equivalent code using the s3fs package in python:

import s3fs
import requests

s3 = s3fs.core.S3FileSystem(profile = 'pete')
s3.ls('my-bucket')

I get this error:

Traceback (most recent call last):

  File "C:\ProgramData\Anaconda3\lib\site-packages\s3fs\core.py", line 504, in _lsdir
    async for i in it:

  File "C:\ProgramData\Anaconda3\lib\site-packages\aiobotocore\paginate.py", line 32, in __anext__
    response = await self._make_request(current_kwargs)

  File "C:\ProgramData\Anaconda3\lib\site-packages\aiobotocore\client.py", line 154, in _make_api_call
    raise error_class(parsed_response, operation_name)

ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied


The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "<ipython-input-9-4627a44a7ac3>", line 5, in <module>
    s3.ls('ma-baseball')

  File "C:\ProgramData\Anaconda3\lib\site-packages\s3fs\core.py", line 993, in ls
    files = maybe_sync(self._ls, self, path, refresh=refresh)

  File "C:\ProgramData\Anaconda3\lib\site-packages\fsspec\asyn.py", line 97, in maybe_sync
    return sync(loop, func, *args, **kwargs)

  File "C:\ProgramData\Anaconda3\lib\site-packages\fsspec\asyn.py", line 68, in sync
    raise exc.with_traceback(tb)

  File "C:\ProgramData\Anaconda3\lib\site-packages\fsspec\asyn.py", line 52, in f
    result[0] = await future

  File "C:\ProgramData\Anaconda3\lib\site-packages\s3fs\core.py", line 676, in _ls
    return await self._lsdir(path, refresh)

  File "C:\ProgramData\Anaconda3\lib\site-packages\s3fs\core.py", line 527, in _lsdir
    raise translate_boto_error(e) from e

PermissionError: Access Denied

I have to assume it's not a config issue within s3 because I can access s3 through the CLI. So something must be off with my s3fs code, but I can't find a whole lot of documentation on profiles in s3fs to figure out what's going on. Any help is of course appreciated.

Solution

import aiobotocore
profile_session = aiobotocore.session.AioSession(profile=profile_name)

s3 = s3fs.S3FileSystem(session=profile_session)

You should get that profile token before assigning using aiobotocore