I have a barologger pressure time series that takes a reading every 15 minutes and a CT2X pressure time series that takes a measurement every 5 minutes. Unfortunately the time series is shifted and they don't align correctly. How can I filter my CT2X time series data so that it matches the time series of the barologger? How could I take an average of the three closest CT2X readings around the barologger time?
I have listed a picture "sample series columns" that ultimately shows what I wish to obtain.
I've tried the index-match tool in excel but I don't think I've used it correctly or if it can be done with this data.
Since you did not show averages as a part of what you expect for results, I chose to return a single closest value by aligning the values by
Barologger Date/Time
and each of the CT2X Date/Time
entries.I added a line and some information to the data you provided:
If you have 365
, this can be done with a complex formula (you will need to use the Microsoft provided Excel Labs add-in to see how the various steps are evaluated):
=LET(
d, Pressures,
bdt, CHOOSECOLS(d, 1),
cdt, CHOOSECOLS(d, 3),
minDif, BYROW(bdt, LAMBDA(arr, MIN(ABS(arr - cdt)))),
allDif, ABS(bdt - TRANSPOSE(cdt)),
pos, BYROW(HSTACK(minDif, allDif), LAMBDA(arr, XMATCH(INDEX(arr, 1), DROP(arr, , 1)))),
closest, HSTACK(TAKE(d, , 2), CHOOSEROWS(DROP(d, , 2), pos)),
f, FILTER(closest, minDif < TIME(0, 2, 0)),
f
)
Or, this can also be accomplished using Power Query, available in Windows Excel 2010+ and Excel 365 (Windows or Mac)
To use Power Query
Data => Get&Transform => from Table/Range
Home => Advanced Editor
Applied Steps
to understand the algorithmM-Code
let
Source = Excel.CurrentWorkbook(){[Name="Pressures"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Barologger Date/Time", type datetime}, {"Barologger LEVEL", type number}, {"CT2X Date/Time", type datetime}, {"CT2X Pressure", type number}}),
#"Added Custom" =Table.AddColumn(#"Changed Type","Match CT2X", (r)=>
let
//generate a list of differences between each of the times
difs = List.TransformMany(
#"Changed Type"[#"CT2X Date/Time"],
each {_ -r[#"Barologger Date/Time"]},
(a,b)=> Number.Abs(Number.From(b))),
//determine the row numbers with the matching min times, using 2 minutes (2/1440) as the threshold
pos = if List.Min(difs) < 2/1440
then List.PositionOf(difs,List.Min(difs))
else null
in pos, Int64.Type),
#"Filtered Rows" = Table.SelectRows(#"Added Custom", each [Match CT2X] <> null and [Match CT2X] <> ""),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"CT2X Date/Time", "CT2X Pressure"}),
#"Matching CT2X" = Table.TransformColumns(#"Removed Columns",{"Match CT2X",
each Table.SelectColumns(#"Added Custom",{"CT2X Date/Time", "CT2X Pressure"}){_},type [#"CT2X Date/Time"=datetime, CT2X Pressure=number]}),
#"Expanded Match CT2X" = Table.ExpandRecordColumn(#"Matching CT2X", "Match CT2X", {"CT2X Date/Time", "CT2X Pressure"}, {"CT2X Date/Time", "CT2X Pressure"})
in
#"Expanded Match CT2X"
Either method gives the following results from the data I posted: