s3fs

Use s3fs only to upload new files, don't care about existing ones already on bucket


I was hoping to use s3fs to upload new files into S3. On the documentation I saw that it doesn't work well when there are multiple clients uploading/syncing to the same bucket.

I really don't care about syncing files from to bucket to my local drive, I only want to perform the opposite: only upload to s3 new files as they are created.

Is there a way to achieve that with s3fs? It wasn't clear on the docs if they offer that functionality by the usage of flags.


Solution

  • s3fs does not synchronize files. Instead it intercepts the open, read, write, etc. calls and relays them to the S3 server. Thus it will work for your upload-only use case. Note that s3fs does use some temporary storage to stage the upload.