Python paho MQTT loop_forever(): how to redirect output to a file while the script is running?

I'm running a script to subscribe to topics of an MQTT broker and fetch the data associated to them. I run the script like this:

$ python3 test_mqtt_client.py

import paho.mqtt.client as paho
import ssl
import random

from config import BROKER_ADDRESS, PORT, CLIENT_CERT, CLIENT_KEY, CA_KEY, SUBSCRIBED_TOPICS, DEST_FOLDER

#"on_message" callback
def on_message(client, userdata, message):
  
  print("received message =",str(message.payload.decode("utf-8")))


  filename = str(random.randint(0,4294967295))

  file = open(DEST_FOLDER + filename + '.json', 'w+')
  file.write(str(message.payload.decode("utf-8")))
  file.close()


client=paho.Client() 
client.on_message=on_message

print("connecting to broker")

client.tls_set(
    CA_KEY,
    certfile=CLIENT_CERT,
    keyfile=CLIENT_KEY,
    cert_reqs=ssl.CERT_REQUIRED,
    tls_version=ssl.PROTOCOL_TLSv1_2,
    ciphers=None
)

client.tls_insecure_set(True)

client.connect(BROKER_ADDRESS, PORT, 60)

for x in SUBSCRIBED_TOPICS:
  client.subscribe(x)
  print('Subscribed to topic "' + x + '".')

client.loop_forever()

time.sleep(1)

The problem: if I try to output to a file like this:

$ python3 test_mqtt_client.py >> /tmp/test_mqtt_client.log

I don't get any content on the file untill I interrupt the script using Ctrl+C.

How can I get the output of the script inside /tmp/test_mqtt_client.log while the script is running? I mean, before interrupting it.

Solution

By default, output to stdout is buffered: that means it's not actually flushed to a file until the output buffer is full, which in turns means that when redirecting stdout to a file, you often won't see anything in the file until your code generates sufficient output to overflow the output buffer.

There are various ways of dealing with with this behavior:

Just output to stderr instead of stdout. stderr is not buffered by default and you should see output immediately. This is the default behavior of e.g. the logging module, which is generally a better idea than print for logging output from long-running programs. You can, of course, just set file=sys.stderr in your print command instead:
```
print("this is a test", file=sys.stderr)
```
(If you do this, remember to redirect stderr instead of stdout on the command line.)
You can run Python in unbuffered mode by adding the -u flag to the command line:
```
python -u test_mqtt_client.py >> test_mqtt_client.log
```
You can explicitly flush the output buffer after each print statement:
```
print("some log message")
sys.stdout.flush()
```