[SOLVED] python Popen stderr=PIPE 6000+ lines causes hang

python Popen stderr=PIPE 6000+ lines causes hang

I am running a command line application via Popen in OSX which produces many lines of command line readout, passing this via stderr to my wx.TextCtrl window. I am using threads and classes and everything has been working fine, until I run something which produces thousands of lines, at which point my application runs good until it hangs at line 6498 and i'm not sure why, any ideas please?

self.process1 = Popen(shlex.split(command), shell=False, stderr=PIPE)
while True:
    line = self.process1.stderr.readline().decode('utf-8')
    wx.CallAfter(self.frame.running_log1.AppendText, line)
    self.process1.stderr.flush()
    if "Some text" in line:
        break

Solution

wx.TextCtrl has an maximum number of characters that can be added to the control. It is platform specific and depends on the native control, but it is documented that it should be at least 32k.

You can try setting a specific maximum length, or delegating that to the native control by setting the max length to 0. See the SetMaxLength() method.

You can probably work out what the current max length is set to by checking the number of bytes contained in your 6000+ lines. Also, you could register an event handler for the wx.EVT_TEXT_MAXLEN(id, func) event which will notify your application tries to exceed the limit.

This is all based on the wxWidgets/wxPython documentation, not on actual experience.

BTW, self.process1.stderr.flush() does nothing as you are not writing to your subprocess.