I am uploading almost 7 TB of files and folders from my remote server to the s3 bucket but I can not see any files on the s3 bucket. only a few files I can see on s3 that was copied successfully.
I have one ec2 server on which I have mounted an s3 bucket using this link
on the remote server, I am using the following script. I have also tested this script and it was working fine for the small size of files
rsync -uvPz --recursive -e "ssh -i /tmp/key.pem" /eb_bkup/OMCS_USB/* appadmin@10.118.33.124:/tmp/tmp/s3fs-demo/source/backups/eb/ >> /tmp/log.txt &
The log file I am generating is showing me files are being copied and all the relevant information like transfer speed, filename, etc. But on the s3 bucket, I can not see any file after the 1st one is copied.
Each file size is from 500MB to 25GB.
Why I cannot see these files on S3?
So the issue I was facing that rsync was copying files on target ec2 and first creating a temporary file and then writing it over S3 bucket. So multiple rsync job was running and the local ebs volume storage on EC2 server was full. That is why rsync wasn't able to create temp files and was kept copying/writing on the socket.