I am trying to use pexpect to have a main processing loop that sends input to a number of other processes in order.
Code
pexpect_test.py
if __name__ == '__main__':
count = 3
processes = []
for x in range(count):
processes.append(pexpect.spawn('/bin/bash', logfile=sys.stdout.buffer))
print(f'Processes Created: {len(processes)}')
inputs = ['a', 'b', 'c']
for input in inputs:
print(f'Processing input: {input}')
for process in processes:
line = f'python test.py {input}'
print(f'Sending Line: {line}')
process.sendline(line)
print('reading')
res = process.read()
print(f'res: {res}')
test.py
if __name__ == '__main__':
print(f'Process Started: {argv[1]} ')
sleep(2)
print(f'Process Complete: {argv[1]}')
Results
$ python pexpect_test.py
Processes Created: 3
Processing input: a
Sending Line: python x.py a
python x.py a
reading
$ python x.py a
Process Started: a
Process Complete: a
$ ^CTraceback (most recent call last):
It looks like the process ends up just waiting at the spawned shell so I either have to use control
+c
or let it timeout. How could I leave the shells running while returning the flow of logic back to the main script?
Simplified Example
if __name__ == '__main__':
ps = pexpect.spawn('/bin/bash', logfile=sys.stdout.buffer)
ps.sendline('python x.py 1 a')
# Expect EOF
ps.expect(pexpect.EOF)
# - The child bash shell stays active and control does not go back to python code
# - Will eventully timeout as expected and raise an exception
# - Does correctly call python script and waits for the defined amount of time
# Expect terminal character
ps.expect_exact('$ ')
# - The wait is skipped but the child shell does not remain active
# Expect literal string printed in child process
ps.expect_exact('Completed')
# - Correctly spawns child thread and waits specified duration
# - Return control back to script
# - The requirement to add a print statement containing unique string is not desirable.
# -There could be hundreds of different processes that need to be run and my group does not manage their code.
# Expect the terminal character twice
ps.expect('\$.*\$ ')
# - For some reason this works exactly as I need it to.
Notes:
subprocess.run(...)
: Not performant enough. The above sample code uses bin/bash
as a placeholder, the actual command I need to run takes 2-3 seconds to initialize and I have a lot of inputs to process(10000+).I was able to determine what pattern to expect with the following approach:
if __name__ == '__main__':
ps = pexpect.spawn('/bin/bash', logfile=sys.stdout.buffer, timeout=7)
ps.sendline('python x.py 1 a')
try:
ps.expect('something')
except:
print(ps.before)
Since the string passed to expect will match nothing, this will result in all output being in the before property which for my local machine was:
b'python x.py 1 a\r\n(base) \x1b]0;name@name-VirtualBox: ~/path/to/src\x07\x1b[01;32mname@name-VirtualBox\x1b[00m:\x1b[01;34m~/path/to/src\x1b[00m$ python x.py 1 a\r\nProcess 1 Started: a\r\nProcess 1 Completed: a\r\n(base) \x1b]0;name@name-VirtualBox: ~/path/to/src\x07\x1b[01;32mname@name-VirtualBox\x1b[00m:\x1b[01;34m~/path/to/src\x1b[00m$ '
This was different on the actual platform I need to use this for but the process will allow you to determine what you need to expect. For my local testing it was ps.expect('\$.*\$ ')
but for the actual platform it was ps.expect('a-unique-string.*# ')