pyfilesystem

Opening a file in pyfilesystem's s3 filesystem fails

I'm trying to simply open a file in pyfilesystem's s3 filesystem as such:

>>> s3fs.listdir('dfparks/test')
['test.txt']
>>> s3fs.open('dfparks/test/test.txt')

I get the error:

fs.errors.ResourceNotFound: resource 'dfparks/test/test.txt' not found

I've tried every variant of the path / s3 url I can think of but I always get the not found error.

Solution

  • fs.s3fs by default works in strict mode, checking for existence of directory markers (keys ending with "/"). You can turn off strict mode by passing strict=False to S3FS constructor or append "strict=0" to url passed to fs.open_fs

    But, unfortunately, S3FS still does these checks in few places even in strict=False mode. There is my PR waiting for review fixing that: https://github.com/PyFilesystem/s3fs/pull/60. In the meantime you can install fixed version using fs-s3fs-ng package:

    pip install fs-s3fs-ng

    Additionally it enables non-strict mode by default and turns on S3 streaming (other PR waiting for acceptance)

    Please let me know if this works for you.