pythonsshsftpparamiko

Closing file after SFTP upload still makes client retry the same upload in `close` sftp_response from paramiko

I run a custom SFTP Server using Paramiko that receives video files from NVR cameras and uploads it to a cloud storage. The flow is simple: The client (NVR) connects and authenticates, we write the incoming file locally, then push it to Google Cloud Storage, and remove the local copy. After that the connection should close and the NVR should stop trying to resend the same file. What I'm seeing is that instead some NVRs keep retrying the exact same upload in a loop right after we delete the local file, which floods the server with repeated writes.

Here is a reduced version of the handler that closes the connection right after upload:

import os, logging, paramiko

class SFTPFileHandle(paramiko.SFTPHandle):
    def __init__(self, flags, upload_file, file_name, file_path, rename_file) -> None:
        super().__init__(flags)
        self.upload_file = upload_file
        self.file_name = file_name
        self.file_path = file_path
        self.rename_file = rename_file

    def chattr(self, path, attr):
        pass

    def stat(self):
        return paramiko.SFTPAttributes.from_stat(os.fstat(self.readfile.fileno()))

    def close(self):
        try:
            super().close()

            try:
                exists = os.path.exists(self.file_path)
                size = os.path.getsize(self.file_path) if exists else -1
                logging.info(f"[Close] path='{self.file_path}' exists={exists} size={size}")
            except Exception as e:
                logging.warning(f"[Close] stat failed for '{self.file_path}': {e}")

            fname = self.rename_file(file_name=self.file_name, file_path=self.file_path)

            if fname:
                if fname == "camera_off":
                    try:
                        os.remove(self.file_path)
                        logging.info("[Cleanup] removed local file (camera_off)")
                    except Exception as e:
                        logging.warning(f"[Cleanup] remove failed: {e}")
                else:
                    ok = self.upload_file(fname, self.file_path)
                    if ok:
                        try:
                            os.remove(self.file_path)
                            logging.info("[Cleanup] removed local file after upload")
                        except Exception as e:
                            logging.warning(f"[Cleanup] remove failed after upload: {e}")
                    else:
                        logging.error("[Close] upload failed")
            else:
                logging.error("[Close] failed to build final name")

        except Exception:
            logging.exception("[Close] error while finalizing")

I expected that once close() finished without raising any errors, the client would see the transfer as successful and stop retrying. Instead, the NVR keeps attempting the upload immediately and we observe duplicate uploads of the same file. Server logs do not show a protocol error and the upload to GCS happens successfully, followed by the removal of the local file.

This suggests my client didn't receive a clear success signal on close...

PS: If anyone wonders, like I wondered myself, unfortunately, no, it's not possible to access logs on the NVR to see what is happening after the upload is complete. The NVRs are potato cameras with little to no management at all, with embedded software that only allows to change the DNS of the SFTP Server and no other option available.

Solution

  • Cannot it be that the close takes too long and the client times out? If you remove the uploading part (or any that takes too long), won't the client stop retrying?

    If that helps, you have these options to make close quicker: