(Note: I am mostly new at Python so my code is more functional than elegant, or so I hope it becomes functional.)
Using Python 3.6 and pygal 2 I am attempting to read in json formated data from a storage array's REST API. I want to pull out the latency values (read, write, total) and create a scatter plot showing the values.
I started with the pygal docs about plotting dates, here.
The timestamps in json are all in epoch milliseconds. So as I read them in, I have to divide to make them work with datetime library.
Sample json data section:
[
1532908042508,
{
"latency.total": 1.09258,
"partialBlocksRatio.total": 18.58528,
"iops.read": 5984.2,
"latency.read": 1.1011,
"iops.write": 2181.4,
"throughput.read": 1461.03762,
"throughput.write": 105.14331,
"throughput.total": 1566.18092,
"iops.total": 8165.6,
"latency.write": 1.06919
}
],
My issue is once I get the data is how to feed it into pygal. I start with defining the chart (taken from the pygal doc example)
I pull the data, convert the epoch milliseconds to seconds, create the time string to be in the format needed (Y, m, d, H, M, S) and put that list in a list. Next load the latency data into a list. When all the data is loaded, I add to pygal and attempt to render to a file.
if key.lower() == 'history':
for key2, historyData in value:
epochSec = key2 / 1000.0 # The key for each history entry is epoch time in milliseconds. Time functions in Python
# appear to want seconds. So the division is needed.
timeStamp = [datetime.fromtimestamp(epochSec).strftime('%Y, %m,
%d, %H, %M, %S')] # Create the list format used for pygal
timeStamps.append(timeStamp)
#print(timeStamp)
for key3 in historyData:
if key3.lower() == 'latency.read':
perfHistory.append(historyData[key3])
for stamp in timeStamps:
#print(stamp[0])
xy_dateLine.add("", datetime(int(stamp[0])),perfHistory)
xy_dateLine.render_to_file('scatter4-1.svc')
the error is Traceback (most recent call last): File "C:/Users/../python/XIO_DataPlotting5.py", line 57, in xy_dateLine.add("", datetime(int(stamp[0])),perfHistory) ValueError: invalid literal for int() with base 10: '2018, 07, 22, 18, 02, 07'
My feeling here is I am overlooking something simple, yet I am at a lost at what. Am I over complicating this? Is there a tutorial for pygal that my google-fu uncovering?
I've just spent about an hour trying to solve a closely related issue – so despite the fact that this is old, I thought that I'd share my solution...
Despite the fact that PyGal documentation doesn't make it terribly clear – PyGal does in fact directly support time/dates in UNIX epoch time...
So for example to plot three points (picking numbers I was playing with for something else) in 2014 – then the code is just:
dateline.add('Epoch Time', [
(1389830399, 400),
(1402826470, 450),
(1420029285, 500)])
Or for the full example – it'd be:
import pygal
from datetime import date
dateline = pygal.DateLine(x_label_rotation=45, range=(0, 750))
dateline.x_labels = [
date(2014, 1, 1),
date(2014, 3, 1),
date(2014, 5, 1),
date(2014, 7, 1),
date(2014, 9, 1),
date(2014, 11, 1),
date(2014, 12, 31)
]
dateline.add('Time', [
(1389830399, 400),
(1402826470, 450),
(1420029285, 500)])
dateline.render_to_png("test.png")
Which gives you a graph that looks like this:
Anyway, I hope that's of use to save someone else the same headache that I've just had! :-)