I am trying to show real time voltage data using a dynamic graph that shows the recorded voltage over time and display the plot inside of a Tkinter window that has other widgets. This program also has to take various actions with switches and relays based on the measured values as well as sending out notifications of these actions via text messages and emails. After many iterations on different methods, I settled on Matplotlib using clear() and draw() in each loop to update the plot and place just the email sending into a sub-process so that the internet delay does not stop the voltage sampling for a long period of time. It is working quite well plotting 3 traces of 500 points each at about 1/4 second plot update rate on a Raspberry Pi 4.
However, as I let the program run, I am finding that the loop time is getting longer and longer, slowing down the loop time from 1/4 second to 2.5 seconds after 16 hours. Also, the Virtual Memory size has grown from 105MB to 500MB.
I added code to isolate the culprit and narrowed it down to the clear() call to Matplotlib. Here is a plot showing the time it took for each component of the loop over 3 hours of the program running in a while loop.
Time components of each loop over time
Here you can see that the time it took for the call to clear() (red line) increased from 0.055 seconds to 0.7 seconds over 3 hours of the loop running. All the other components of the loop stayed pretty much constant except for the calls to plot() having a couple of big spikes. But the time taken by the clear() call keeps on increasing and is not acceptable. It is most likely the culprit for the memory leakage too.
I have extracted the portion of my program pertaining to this problem in the following program. This can be extracted and ran in python3. It measures the time it takes for the calls to clear(), plot() and draw() in each loop of the while loop and plots it dynamically on the screen. You can see that the call to clear is slowly increasing. This program also allows you to see the effect of other activities in the system on the time to execute these calls. You can see that just moving the mouse around has an effect. Try open the browser or play a video.
import time
from time import sleep
from datetime import datetime
import math # provides math functions
from tkinter import * # provides GUI capability
from tkinter import ttk
from tkinter import messagebox
import tkinter as tk
import matplotlib.pyplot as plt
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg
# Set root window to center of screen. Parameters are size of window itself.
def center_window(self, width=300, height=200):
# get screen width and height
screen_width = self.winfo_screenwidth()
screen_height = self.winfo_screenheight()
# calculate position x and y coordinates
x = (screen_width/2) - (width/2)
y = (screen_height/2) - (height/2)
self.geometry('%dx%d+%d+%d' % (width, height, x, y))
def alldone(*args):
global running
running = False
root = Tk() # Create base window class as root
root.wm_title("Green Canyon APFM System Ver: 0.6")
center_window(root,1024,580)
running = True
root.protocol("WM_DELETE_WINDOW", alldone) # Define routine to call if window closes
time_plot = []
clear_plot = []
plot_plot = []
draw_plot = []
figure3 = plt.Figure(figsize=(9.9,5.8), dpi=100)
ax3 = figure3.add_subplot(111)
ax3.plot(time_plot,clear_plot,"r-", label="clear() call") # Red line
ax3.plot(time_plot,plot_plot,"g-", label="plot() calls") # Green line
ax3.plot(time_plot,draw_plot,"b-", label="clear() call") # Blue line
scatter3 = FigureCanvasTkAgg(figure3, root)
scatter3.get_tk_widget().grid(column=0, row=0, rowspan=2)
ax3.legend(loc=6)
ax3.set_xlabel('Time (secs)')
ax3.set_ylabel('Task time (sec) for the calls')
ax3.set_title(' EndTime = '+datetime.now().strftime("%H:%M:%S"))
ax3.grid()
scatter3.draw()
loopclock = time.time()
pclock = 0.0
"""-------------------------------------------------------------------------------
Main Loop
-------------------------------------------------------------------------------"""
t2=t3=t4=t5=t6=t7=t8=0.0
t2a=t3a=t4a=t5a=t6a=t7a=t8a=0.0
nn = 0
while running:
c2 = time.time()
"""----------------------------------------------------------------------
This segment update the plot on the screen
----------------------------------------------------------------------"""
ax3.clear()
c3 = time.time()
t2 = c3 - c2
ax3.plot(time_plot,clear_plot,"r-", label="clear() call") # Red line
ax3.plot(time_plot,plot_plot,"g-", label="plot() calls") # Green line
ax3.plot(time_plot,draw_plot,"b-", label="draw() call") # Blue line
c4 = time.time()
t3 = c4 - c3
ax3.legend(loc=6)
c5 = time.time()
t4 = c5 - c4
ax3.set_xlabel('Time (secs)')
ax3.set_ylabel('Voltage (V)')
c6 = time.time()
t5 = c6 - c5
looptime = time.time() - loopclock
loopclock = time.time()
ax3.set_title(' EndTime = '+datetime.now().strftime("%H:%M:%S")+
" LT="+f"{looptime:.2f}"+
f"\n {nn:4d}|{t2:.3f}|{t3:.3f}|{t4:.3f}|{t5:.3f}|{t6:.3f}|{t7:.3f}|{t8:.3f}")
ax3.grid()
c7 = time.time()
t6 = c7 - c6
scatter3.draw()
c8 = time.time()
t7 = c8 - c7
root.update()
c9 = time.time()
t8 = c9 - c8
# print out the max values in every 15 second intervals for plotting
t2a = max(t2,t2a)
t3a = max(t3,t3a)
t4a = max(t4,t4a)
t5a = max(t5,t5a)
t6a = max(t6,t6a)
t7a = max(t7,t7a)
t8a = max(t8,t8a)
nn += 1
if time.time() > pclock + 15.0:
pclock = time.time()
print(f"{nn:5d},{t2a:.4f}, {t3a:.4f}, {t4a:.4f}, {t5a:.4f}, {t6a:.4f}, {t7a:.4f}, {t8a:.4f}")
t2a=t2
t3a=t3
t4a=t4
t5a=t5
t6a=t6
t7a=t7
t8a=t8
xtime = (time.time() + 2209132800) % 60.0
if len(time_plot) >= 500:
time_plot.pop(0)
clear_plot.pop(0)
plot_plot.pop(0)
draw_plot.pop(0)
if len(time_plot) > 0 and time_plot[-1] > xtime: # If we are rolling over to the next minute, decrease all the old values by 1 minute
for j in range(len(time_plot)):
time_plot[j] -= 60.0
time_plot.append(xtime)
clear_plot.append(t2)
plot_plot.append(t4)
draw_plot.append(t7)
root.quit()
sys.exit()
Would much appreciate any pointers on how I can fix this CPU and memory leak. I found suggestions on making the plotting in a separate process so that all the memory is retrieved when the task is terminated. But terminating the task that plots to the screen every loop may blank the display or make it flicker. Wondering if there is a better way to do dynamic plotting using python and matplotlib. I am using python 3.7 and Matplotlib 3.0.2.
Problem resolved. I installed Matpoltlib 3.1.4 and the problem went away.