Substitution of subprocess.PIPE in Python?

I am using subprocess module to interact with output of the linux commands. below is my code.

import subprocess
import sys

file_name = 'myfile.txt'
p = subprocess.Popen("grep \"SYSTEM CONTROLLER\" "+ file_name, stdout=subprocess.PIPE, shell=True)
(output, err) = p.communicate()
print output.strip()

p = subprocess.Popen("grep \"controller\|worker\" "+ file_name, stdout=subprocess.PIPE, shell=True)
(output, err) = p.communicate()
lines = output.rstrip().split("\n")
print lines

My program hangs while executing second subprocess i.e.

p = subprocess.Popen("grep \"controller\|worker\""+ file_name,stdout=subprocess.PIPE, shell=True)

I got to know that the reason of process hang is buffer redirected to subprocess.PIPE is getting filled, which blocks the process from writing further.

I want to know if there is any way to avoid the buffer full situation so that my program keeps on executing without any hang issue ?

Solution

The actual issue is that there is a whitespace missing between the pattern and the filename and therefore grep waits for input on the standard input (stdin).

"buffer full" (.communicate() is not susceptible) or p.stdout.read() (it fixes nothing: it loads the output into memory and unlike .communicate() it fails if more than one pipe is used) are a red herring here.

Drop shell=True and use a list argument for the command:

#!/usr/bin/env python
from subprocess import Popen, PIPE

p = Popen(["grep", r"controller\|worker", file_name], stdout=PIPE)
output = p.communicate()[0]
if p.returncode == 0:
    print('found')
elif p.returncode == 1:
    print('not found')
else: 
    print('error')