I have duplicity running for years, stable and without problems. Now I encountered the following problem when I backup my files to Amazon S3.
All files in the format "duplicity-full.YYYYMMDD1ThhmmssZ.volX.difftar.gpg" are written to Amazon S3 successfully. At least I can see them in the directory/bucket.
Unfortunately, when uploading the "duplicity-full-signatures.YYYYMMDD1ThhmmssZ.sigtar.gpg" an error occurs stating:
Backtrace of previous error: Traceback (innermost last):
File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 387, in inner_retry
return fn(self, *args)
File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 574, in move
self.__do_put(source_path, remote_filename)
File "/usr/lib/python3/dist-packages/duplicity/backend.py", line 544, in __do_put
self.backend._put(source_path, remote_filename)
File "/usr/lib/python3/dist-packages/duplicity/backends/_boto_single.py", line 265, in _put
self.upload(source_path.name, key, headers)
File "/usr/lib/python3/dist-packages/duplicity/backends/_boto_single.py", line 319, in upload
key.set_contents_from_filename(filename, headers,
File "/usr/local/lib/python3.10/dist-packages/boto/s3/key.py", line 1375, in set_contents_from_filename
return self.set_contents_from_file(fp, headers, replace, cb,
File "/usr/local/lib/python3.10/dist-packages/boto/s3/key.py", line 1307, in set_contents_from_file
self.send_file(fp, headers=headers, cb=cb, num_cb=num_cb,
File "/usr/local/lib/python3.10/dist-packages/boto/s3/key.py", line 760, in send_file
self._send_file_internal(fp, headers=headers, cb=cb, num_cb=num_cb,
File "/usr/local/lib/python3.10/dist-packages/boto/s3/key.py", line 957, in _send_file_internal
resp = self.bucket.connection.make_request(
File "/usr/local/lib/python3.10/dist-packages/boto/s3/connection.py", line 667, in make_request
return super(S3Connection, self).make_request(
File "/usr/local/lib/python3.10/dist-packages/boto/connection.py", line 1070, in make_request
return self._mexe(http_request, sender, override_num_retries,
File "/usr/local/lib/python3.10/dist-packages/boto/connection.py", line 1030, in _mexe
raise ex
File "/usr/local/lib/python3.10/dist-packages/boto/connection.py", line 939, in _mexe
response = sender(connection, request.method, request.path,
File "/usr/local/lib/python3.10/dist-packages/boto/s3/key.py", line 856, in sender
http_conn.send(chunk)
File "/usr/lib/python3.10/http/client.py", line 999, in send
self.sock.sendall(data)
File "/usr/lib/python3.10/ssl.py", line 1266, in sendall
v = self.send(byte_view[count:])
File "/usr/lib/python3.10/ssl.py", line 1235, in send
return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2426)
Attempt of move Nr. 1 failed. SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2426)
After 5 attemps, the upload is stopped. Checking the upload online monitor of my router I see that a file is uploaded, but only for short time (like 10 sec). In addition, I cannot find the "duplicity-full-signatures.YYYYMMDD1ThhmmssZ.sigtar.gpg" in the S3 directory/bucket. This problem only occurs with one big directory (~1 TB). Other directories work without any problems (same user, same password, same keys etc.)
I have Ubuntu 22.04.4 LTS with kernel 5.15.0-116-generic with duplicity 0.8.21.
Any ideas? How to debug? And how to find the problem? I got the error message running
duplicity --verbosity d
sounds like you hit the upper file size limit of Amazon S3 for single put operations (5GB). unfortunately the manifest is not split to volume sizes even in recent duplicity.
but, as Amazon S3 documentation states files (objects) up to 5TB can be uploaded with multipart upload API, which is the default in current duplicity 3.x as it uses the current boto3 python module.
your old duplicity already had this backend integrated, albeit only as alternative only. you can use it using the boto3+s3:// target url scheme. you will need to have package python3-boto3 installed.