How to make pexpect wait for full execution?

I'm using Python with pexpect to automate a script that builds a Docker image. The script takes about 20 minutes to complete. Here's my current code:

child.sendline(f"./create_package_image.sh test-user -t rhel8; echo COMMAND_COMPLETED_MARKER")

# Pexpect doesn't wait for this and continues the script
child.expect("COMMAND_COMPLETED_MARKER", timeout=1800)

# Pexpect doesn't wait for this and continues the script
child.expect("test-user@", timeout=1800)

The problem is that pexpect is not waiting for the COMMAND_COMPLETED_MARKER or my ssh promt ($, test-user@, ...) to appear; it immediately returns true and the script continues.

The Docker build script runs successfully.

What's the most reliable way to make pexpect wait for the completion of a long-running command? Are there alternatives to using a custom marker?

Edit: Here is a minimalistic example: Bash File run_5_secs.sh

echo "Script started "
sleep 5
echo "Script completed"

My Python script:

import pexpect
import time

start_time = time.time()
child = pexpect.spawn("ssh host")
child.sendline("./run_5_secs.sh; echo 'COMPLETED'")
child.expect("$")
child.expect("COMPLETED")

end_time = time.time()

elapsed_time = end_time - start_time
print(f"Command executed in {elapsed_time:.2f} seconds")

Execution time is about 0.06 seconds -> not waiting for run_5_secs.sh file to finish

Solution

expect("$") always succeeds because the param is handled as an RE pattern. You should use a more specific pattern to match the shell prompt.
expect() would try to match against everything displayed to the tty which usually includes both the command itself and the command's output. So child.expect("COMPLETED") actually matched the COMPLETED in the command ./run_5_secs.sh; echo 'COMPLETED' rather than in its output. To match the output you can change the command a little bit so the command itself would not include COMPLETED.

Example foo.py:

import pexpect, sys

re_prompt = 'bash-[.0-9]+[$#] '

child = pexpect.spawn('bash --norc', encoding='utf8')
child.logfile_read = sys.stdout

child.expect(re_prompt)

child.sendline('echo COMPLETED')
child.expect('COMPLETED') # this matches the 'COMPLETED' in 'echo COMPLETED'
child.expect('COMPLETED') # this matches the 'COMPLETED' in the output
child.expect(re_prompt)

child.sendline('echo C""OMPLETED') # the command itself does not include COMPLETED
#                     ^^
child.expect('COMPLETED')          # so this matches the 'COMPLETED' in the output
child.expect(re_prompt)

child.sendline('exit')
child.expect(pexpect.EOF)

Test it:

$ python3 foo.py
bash-5.3# echo COMPLETED
COMPLETED
bash-5.3# echo C""OMPLETED
COMPLETED
bash-5.3# exit
exit
$