[SOLVED] how to maintain the order of logs when using python's subprocess to call other processes

how to maintain the order of logs when using python's subprocess to call other processes

I have a.sh as below

#!/bin/bash

echo 100
python3 b.py
if [ $? -ne 0 ]; then
    exit 1
fi
echo "this will not be printed"

and b.py as below

import subprocess

print("101")
result = subprocess.run(["bash", "c.sh"], capture_output=True, text=True)
print(result.stderr)
print(result.stdout)
if result.check_returncode:
    raise Exception("")

and c.sh as below:

#!/bin/bash

echo 102 >&2
echo 103
echo 104 >&2
exit 1
echo "this will not be printed"

In a bash terminal I run bash a.sh and the result is

100
101
102
104

103

Traceback (most recent call last):
  File "/Users/abamavad/ssc-github/ssce/cmn-deployment/b.py", line 9, in <module>
    raise Exception("")
Exception

I am looking for a way to keep the order of the logs so that I get

100
101
102
103
104

Traceback (most recent call last):
  File "/Users/abamavad/ssc-github/ssce/cmn-deployment/b.py", line 9, in <module>
    raise Exception("")
Exception

I know where my problem is:

result = subprocess.run(["bash", "c.sh"], capture_output=True, text=True)
print(result.stderr)
print(result.stdout)

but don't know how to fix it. how should I maintain the order of the log whether or not they are stdout or stdin

Solution

if you try to read stdout and stderr then by the time you read them they will already be garbled, instead you need to merge both stdout and stderr at the operating system level, by passing stdout=subprocess.PIPE, stderr=subprocess.STDOUT.

result = subprocess.run(["bash", "c.sh"], stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT, text=True)
print(result.stdout)  # both stdout and stderr are merged