I'm trying to write a small application GUI (GTK) application with Python 3.9 and PyGObject that reads from a log file and displays the data in a GUI window.
In this particular case I'm trying to continuously read from the systemd Journal, which can be done e.g. with journalctl -f via the terminal.
For a CLI app I saw one could use something like this to pipe the output of the log file to the STOUT of the python script:
p = Popen(["journalctl", "--user-unit=appToMonitor"], stdout=PIPE)
with p.stdout:
for line in iter(p.stdout.readline, b''):
print(line, end=''),
p.wait()
OFC for a GUI app I need to somehow hook into the GTK main loop so the above solution does not work. I did some googling and from what I can see a solution is to use gobject.timeout_add() to regularly poll the log file for changes.
Is this the best way to get data from a system log file or is there another solution that can be used to avoid polling? If gobject.timeout_add() is the way to go, how can I ensure that I'll be only adding the log lines that have been added since the last read from the log file?
It's not necessary to use gobject.timeout_add(); you can just use the threading module, which specializes in threads, and is simpler than trying to use GTK.
The nice thing about using threading is that, even though it's not part of the Gtk group of modules, it still doesn't block Gtk's main loop. Here is a simple example of a simple timeout thread using threading:
import threading
def function():
print("Hello!")
# Create a timer
timer = threading.Timer(
1, # Interval (in seconds; can be a float) after which to call the function
function # Function to call after time interval
)
timer.start() # Start the timer
This calls function() once after a period of 1 second, and then exits.
However, in your case, you want to call the function multiple times, to repeatedly check the state of the logs. For this, you can recreate the timer inside function(), and run it again:
import threading
def function():
global timer
print("Hello!")
# Recreate and start the timer
timer = threading.Timer(1, function)
timer.start()
timer = threading.Timer(1, function)
timer.start()
Now to check if any new lines have been added to the log, the program needs to read the log and then compare the lines of the most recent reading to the lines of the previous reading.
First of all, to compare the lines at all, they need to be stored in two lists: one containing the most recently read set of lines, the other containing the previous set of lines. Here is an example of code that reads the output of the log, stores the lines in a list, and then prints the list:
from subprocess import PIPE, Popen
# The list storing the lines of the log
current_lines = []
# Read the log
p = Popen(["journalctl", "--user-unit=appToMonitor"], stdout=PIPE)
with p.stdout:
for line in iter(p.stdout.readline, b''):
current_lines.append(line.decode("utf-8")) # Add the lines of the log to the list
p.wait()
print(current_lines)
Note that line.decode("uft-8") must be used, because the output of p.stdout.readline is in bytes. Use whatever encoding your program uses; utf-8 is just a common encoding.
You can then use the difflib module to compare the two lists. Here is an example program that compares two lists of lines, and prints whatever lines are in the second list and not in the first:
import difflib
# Two lists to compare
last_lines = ["Here is the first line\n"]
current_lines = ["Here is the first line\n", "and here is the second line"]
# Iterate through all the lines, and check for new ones
for line in difflib.ndiff(last_lines, current_lines):
# Print the line only if it was not in last_lines
if line[0] == "+": # difflib inserts a "+" for every addition
print("Line added:", line.replace("+ ", "")) # Remove the "+" from the line and print it
The finale: putting all these concepts into one program that reads the output from the log every second, and adds any new lines to the text widget in the window. Here is a simple Gtk application that does just that:
import difflib
import gi
import threading
gi.require_version("Gtk", "3.0")
from gi.repository import Gtk
from subprocess import PIPE, Popen
class App(Gtk.Window):
"""The application window."""
def __init__(self):
Gtk.Window.__init__(self)
self.connect("delete-event", self.quit)
# The list of lines read from the log file
self.current_lines = []
# The list of previously read lines to compare to the current one
self.last_lines = []
# The box to hold the widgets in the window
self.box = Gtk.VBox()
self.add(self.box)
# The text widget to output the log file to
self.text = Gtk.TextView()
self.text.set_editable(False)
self.box.pack_start(self.text, True, True, 0)
# A button to demonstrate non-blocking
self.button = Gtk.Button.new_with_label("Click")
self.box.pack_end(self.button, True, True, 0)
# Add a timer thread
self.timer = threading.Timer(0.1, self.read_log)
self.timer.start()
self.show_all()
def quit(self, *args):
"""Quit."""
# Stop the timer, in case it is still waiting when the window is closed
self.timer.cancel()
Gtk.main_quit()
def read_log(self):
"""Read the log."""
# Read the log
self.current_lines = []
p = Popen(["journalctl", "--user-unit=appToMonitor"], stdout=PIPE)
with p.stdout:
for line in iter(p.stdout.readline, b''):
self.current_lines.append(line.decode("utf-8"))
p.wait()
# Compare the log with the previous reading
for d in difflib.ndiff(self.last_lines, self.current_lines):
# Check if this is a new line, and if so, add it to the TextView
if d[0] == "+":
self.text.set_editable(True)
self.text.get_buffer().insert_at_cursor(d.replace("+ ", ""))
self.text.set_editable(False)
self.last_lines = self.current_lines
# Reset the timer
self.timer = threading.Timer(1, self.read_log)
self.timer.start()
if __name__ == "__main__":
app = App()
Gtk.main()
The button demonstrates that threading doesn't block execution while waiting; you can click it all you want, even while the program is reading the log!