pythonsubprocessread-write

How to ensure file has been written to disk before reading it


I want to read a plan.txt file generated by the fast-downward planner, to do so I'm using subprocess as follows:

import subprocess

search_options = "lama-first"
## Call Planner
cmd = [
    "python3",
    self.planner_path,
    "--alias",
    search_options,
    "--plan-file",
    self.plan_path,
    self.domain_path,
    self.problem_path,
]
## Read plan.txt
cmd2 = ["cat", self.plan_path]
subprocess.run(cmd, shell=False, stdout=subprocess.PIPE)

plan = []
try:
    # Huge dealy to be sure that the plan is written to the txt file
    time.sleep(1)
    result2 = subprocess.run(cmd2, shell=False, stdout=subprocess.PIPE)
    plan = result2.stdout.decode("utf-8")
except:
    pass

Can you please tell me what is the best approach to be sure that the plan was written to disk before trying to read it, rather than adding huge time delay? thanks in advance.


Solution

  • No need to call time.sleep(...), because (1) subprocess.run waits for the subprocess to exit; (2) the libc and the kernel together make sure that all pending data is flushed to the output files when the (sub)process exits.

    The modified code looks like this:

    import os
    import subprocess
    
    try:
        # Remove output of previous invocation.
        os.remove(self.plan_path)
    except OSError:
        pass
    search_options = "lama-first"
    cmd = [
        "python3",
        self.planner_path,
        "--alias",
        search_options,
        "--plan-file",
        self.plan_path,
        self.domain_path,
        self.problem_path,
    ]
    subprocess.run(cmd).check_returncode()
    with open(self.plan_path, encoding="utf-8") as f:
        plan = f.read()
    

    Some comments about the code in the question:

    FYI Because of kernel buffers and caches, data may be flushed to permanent storage (e.g. filesystem on the HDD or SSD) several seconds later. However, you don't have to wait for that in your program, because the kernel will make sure that any subsequent read following a (flushed) file write will see the output of the write, even if the output is only in buffers or caches.