pythonshutilbandwidth

Python: Shutil, How is a file copied from Source to Destination


I am trying to copy files from one source machine(A) to destination machine(C), where the script that does this transfer is on a different machine (B).

I want to understand if shutil.copyfileobj is copying bits from A -> B -> C or doing a direct transfer from A -> C.

If shutil.copyfileobj is not doing a direct transfer are there any other package functions that can help achieve the implementation that I am looking for.

Example Code:

from pathlib import Path
import shutil

# Current path of the script
print(Path.getcwd())
# C://users/myname/

machine_a = Path("//NETWORK_DRIVE_1/foo")
machine_b = Path("//NETWORK_DRIVE_2/bar")

filename = "biz.txt"

shutil.copyfileobj(
    src = machine_a.joinpath(filename),
    dst = machine_b.joinpath(filename),
    length = 2048 * 1024  # Chunk transfer
)

Solution

  • The code for copyfileobj:

    def copyfileobj(fsrc, fdst, length=0):
        """copy data from file-like object fsrc to file-like object fdst"""
        if not length:
            length = COPY_BUFSIZE
        # Localize variable access to minimize overhead.
        fsrc_read = fsrc.read
        fdst_write = fdst.write
        while True:
            buf = fsrc_read(length)
            if not buf:
                break
            fdst_write(buf)
    

    i.e. it read it into memory on B before writing it to C.. But there is a lot of OS-specific code to make it faster (Lib/shutil.py).

    Look into winrs or psexec for running commands on the remote server itself.