Looking for a more efficient way to loop over and compare datetimeindex values in two Series objects with different frequencies.
Imagine two Pandas series, each with a datetime index covering the same year span yet with different frequencies for each index. One has a frequency of days, the other a frequency of hours.
range1 = pd.date_range('2016-01-01','2016-12-31', freq='D')
range2 = pd.date_range('2016-01-01','2016-12-31', freq='H')
I'm trying to loop over these series using their indexes as a lookup to match days so I can compare data for each day.
Right now I'm using multi-level for loops and if statements (see below); the time to complete these loops seems excessive (5.45 s per loop) compared with what I'm used to in Pandas operations.
for date, val in zip(frame1.index, frame1['data']): # freq = 'D'
for date2, val2 in zip(frame2.index, frame2['data']): # freq = 'H'
if date.day == date2.day: # check to see if dates are a match
if val2 > val: # compare the values
# append values, etc
Is there a more efficient way of using the index in frame1 to loop over the index in frame2 and compare the values in each frame for a given day? Ultimately I want to create a series of values wherever frame2 vals are greater than frame1 vals.
Create two separate series with random data and assign each a datetime index.
import pandas as pd
import numpy as np
range1 = pd.date_range('2016-01-01','2016-12-31', freq='D')
range2 = pd.date_range('2016-01-01','2016-12-31', freq='H')
frame1 = pd.Series(np.random.rand(366), index=range1)
frame2 = pd.Series(np.random.rand(8761), index=range2)
Yes, use resample
, asfreq
and pd.concat
.
Use resample to get the proper frequency out of your series.
asfreq (which sounds sort of dirty) is use to convert back to a series with frequency defined in resample.
Concatenate with frame1 to get values side-by-side.
df = pd.concat([frame1,frame2.resample('1D').asfreq()],axis=1)
df.head()
Output:
0 1
2016-01-01 0.147067 0.235858
2016-01-02 0.820398 0.353275
2016-01-03 0.840499 0.186273
2016-01-04 0.505740 0.340201
2016-01-05 0.547840 0.695041
Then, you can us the following to get back to your series of frame2 exceeding frame1.
df.columns = ['frame1','frame2']
df.query('framed1 < frame2')['frame2']