pythonsubprocessstdoutstderrflush

How is restic outputting data to the screen but not to stdout or stderr?


Updating line

I have a question about where the output of a certain command is going. I'm using restic as an example of a command that behaves this way. It's the last line of the command that's of interest to me, because it updates while the restore process is working.

For context, restic's a back-up utility; the command below is restoring the contents of my-repo from a backup stored on example.com back to my local machine.

$ restic restore latest --repo sftp:matt@example.com:my-repo --target . --verbose
repository ae412afe opened (version 1)
restoring <Snapshot dff2f51a> to /Users/matt
[0:11] 0.02%  192 files 37.687 MiB, total 39644 files 171.389 GiB

The last line updates over time until it reaches 100%, e.g.

[12:31] 1.22%  2989 files 2.087 GiB, total 39644 files 171.389 GiB

Execute from python

I want to run the restore command from a Python script (using subprocess.Popen) and want the script to print out the updating line but haven't been able to get that working.

I suspected it could be that the updating line was being output to stderr instead of stdout. So I repeated the restore command and redirected the stdout and stderr to files, as below:

restic restore lat...rbose > out.log 2> err.log

err.log remains empty, and out.log only contains the restoring <Snapshot dff2f51a> to /Users/matt line, nothing else.


Python snippet

Here's the snippet of my code calling Popen:

backup_command = [
    # fmt: off
    "restic",
    "--repo", repo_name,
    "backup",
    "--verbose",
    "--exclude", "node_modules",
    "--exclude", "whatever",
    # fmt: on
]
backup_process = subprocess.Popen(backup_command, env=env_vars)
backup_process.wait()

In reference to jurez's answer below, I'm not passing stdin or stderr, but having looked at the source for Popen, they default to None, as per jurez's suggestion.


Questions

I have a couple of questions.


Solution

  • From the restic documentation:

    When running from a non-interactive console progress reporting is disabled by default to not fill your logs. For interactive and non-interactive consoles the environment variable RESTIC_PROGRESS_FPS can be used to control the frequency of progress reporting. Use for example 0.016666 to only update the progress once per minute.

    My use of Popen creates a non-interactive shell, resulting in no progress reporting output. Setting the value works as advertised. I find it easier to think of as "seconds between each update" rathen that "frames per second" so use 1 / INTERVAL_IN_SECONDS to set the FPS.

    Credit to jasonharper for looking up the docs, thanks!