pythonpandascomparisondatetimeindextime-frequency

PANDAS - Loop over two datetime indexes with different sizes to compare days and values


Looking for a more efficient way to loop over and compare datetimeindex values in two Series objects with different frequencies.

Setup

Imagine two Pandas series, each with a datetime index covering the same year span yet with different frequencies for each index. One has a frequency of days, the other a frequency of hours.

range1 = pd.date_range('2016-01-01','2016-12-31', freq='D')
range2 = pd.date_range('2016-01-01','2016-12-31', freq='H')

I'm trying to loop over these series using their indexes as a lookup to match days so I can compare data for each day.

What I'm doing now...slow.

Right now I'm using multi-level for loops and if statements (see below); the time to complete these loops seems excessive (5.45 s per loop) compared with what I'm used to in Pandas operations.

for date, val in zip(frame1.index, frame1['data']): # freq = 'D'
    for date2, val2 in zip(frame2.index, frame2['data']): # freq = 'H'
        if date.day == date2.day: # check to see if dates are a match
            if val2 > val: # compare the values
                # append values, etc

Question

Is there a more efficient way of using the index in frame1 to loop over the index in frame2 and compare the values in each frame for a given day? Ultimately I want to create a series of values wherever frame2 vals are greater than frame1 vals.

Reproducible (Tested) Example

Create two separate series with random data and assign each a datetime index.

import pandas as pd
import numpy as np

range1 = pd.date_range('2016-01-01','2016-12-31', freq='D')
range2 = pd.date_range('2016-01-01','2016-12-31', freq='H')

frame1 = pd.Series(np.random.rand(366), index=range1)
frame2 = pd.Series(np.random.rand(8761), index=range2)

Solution

  • Yes, use resample, asfreq and pd.concat.

    Use resample to get the proper frequency out of your series.

    asfreq (which sounds sort of dirty) is use to convert back to a series with frequency defined in resample.

    Concatenate with frame1 to get values side-by-side.

    df = pd.concat([frame1,frame2.resample('1D').asfreq()],axis=1)
    df.head()
    

    Output:

                       0         1
    2016-01-01  0.147067  0.235858
    2016-01-02  0.820398  0.353275
    2016-01-03  0.840499  0.186273
    2016-01-04  0.505740  0.340201
    2016-01-05  0.547840  0.695041
    

    Then, you can us the following to get back to your series of frame2 exceeding frame1.

    df.columns = ['frame1','frame2']
    df.query('framed1 < frame2')['frame2']