amazon-web-servicesdockeramazon-s3sftps3fs

s3fs get file throw Input/output error or Couldn't read from remote file "/mnt/s3/" : Failure


I trying to setup sftp with s3fs but facing accessing errors for files.

I create docker image with mounting S3 bucket in entrypoint, I can ssh\sftp to its container and see files and folder from S3 bucket. But I cant get this files:

sftp:

s     get pin
Fetching /mnt/s3/pin to pin
/mnt/s3/pin                                                                                                  0%    0     0.0KB/s - stalled -
Couldn't read from remote file "/mnt/s3/pin" : Failure

ssh:

sftpusername@1234ebc1e2e3:~/s3$ cat pin
cat: pin: Input/output error

What should be a reason?

Here is entrypoint:

#!/bin/bash

adduser --disabled-password --gecos "" $SFTP_USER

echo "$SFTP_USER:$SFTP_PASSWORD" | chpasswd
echo "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs
mkdir -p /mnt/s3
s3fs $S3_BUCKET /mnt/s3 -o passwd_file=~/.passwd-s3fs -o allow_other -o uid=$(id -u $SFTP_USER)

ln -s /mnt/s3 /home/$SFTP_USER

echo "SFTP user: $SFTP_USER"
echo "S3 Bucket: $S3_BUCKET"
echo "Ready to accept connections..."
exec /usr/sbin/sshd -D

Dockerfile:

FROM ubuntu:16.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get -y update
RUN apt-get -y install openssh-server
RUN apt-get -y install automake autotools-dev g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
RUN git clone https://github.com/s3fs-fuse/s3fs-fuse.git && \
    cd s3fs-fuse && \
    ./autogen.sh && \
    ./configure && \
    make && \
    make install

RUN mkdir -p /var/run/sshd

COPY entrypoint /
RUN chmod +x /entrypoint

EXPOSE 22

ENTRYPOINT ["/entrypoint"]

Build image:

docker build -t sftp/s3fs .

Run container

docker run -p 22 --privileged --env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY --env SFTP_USER --env SFTP_PASSWORD --env SSH_KEY --env S3_BUCKET --env S3_KEY sftp/s3fs

After that I can ssh\sftp to th container (here is a port forwarded from 22, can check with dokcer ps)

sftp -P 32775 sftpusername@localhost

Inside the container I can see s3 bucket with mounted s3 bucket (directories and files). I can create new dir, delete created dir, but unable to get files, existed in the bucket.


Solution

  • Changing the way how to get s3fs fixed the problem.

    I start using apt-get:

    RUN apt-get -y install s3fs
    

    instead of git clone:

    RUN git clone https://github.com/s3fs-fuse/s3fs-fuse.git && \
    cd s3fs-fuse && \
    ./autogen.sh && \
    ./configure && \
    make && \
    make install