pythonfilesshnetwork-programmingrsync

How to write a string to a file on a remote machine?


On Machine1, I have a Python2.7 script that computes a big (up to 10MB) binary string in RAM that I'd like to write to a disk file on Machine2, which is a remote machine. What is the best way to do this?

Constraints:

I'm rooting for an rsync solution, since it is a self-contained entity that I understand reasonably well, and requires no configuration on Machine2.


Solution

  • You open a new SSH process to Machine2 using subprocess.Popen and then you write your data to its STDIN.

    import subprocess
    
    cmd = ['ssh', 'user@machine2',
           'mkdir -p output/dir; cat - > output/dir/file.dat']
    
    p = subprocess.Popen(cmd, stdin=subprocess.PIPE)
    
    your_inmem_data = 'foobarbaz\0' * 1024 * 1024
    
    for chunk_ix in range(0, len(your_inmem_data), 1024):
        chunk = your_inmem_data[chunk_ix:chunk_ix + 1024]
        p.stdin.write(chunk)
    

    I've just verified that it works as advertised and copies all of the 10485760 dummy bytes.

    P.S. A potentially cleaner/more elegant solution would be to have the Python program write its output to sys.stdout instead and do the piping to ssh externally:

    $ python process.py | ssh <the same ssh command>