kqlazure-data-explorer

How to efficiently find if any field changed in a Kusto table?


I have a table that is populated every hour or so with new updates on every disk in the system. I want to isolate two instances of this update and see whether any of the fields for each disk (identified by unique disk id) have changed in this time.

For example the following kusto query shows what I want to achieve (in my problem both T1 and T2 would come from the same table, but this is how I want it to work)

let T1 = datatable(DiskId:int, IsLeased:bool, NumSnapshots:int) [
1, true, 2,
2, false, 0,
3, true, 0,
4, true, 1,
];
let T2 = datatable(DiskId:int, IsLeased:bool, NumSnapshots:int) [
1, true, 0,
2, true, 0,
3, true, 0,
4, true, 1,
];
T1
| join kind = inner (T2) on DiskId
| project DiskId, IsLeasedCheck = IsLeased == IsLeased1, NumSnapshotsCheck = NumSnapshots == NumSnapshots1
| where IsLeasedCheck == false or NumSnapshotsCheck == false

will get me an output as follows, so I can tell at a glance which property if any changed.

DiskId IsLeasedCheck NumSnapshotsCheck
1 true false
2 false true

In my case however, I need to do this for several columns, I wanted to ask whether this can be done in a less tedious way in kusto. I don't need to solve it in exactly this way, I just need to know if any fields changed in any way.


Solution

  • If you just want to figure out if any column has changed, you could merge all columns and compare them:

    let T = datatable(DiskId:int, IsLeased:bool, NumSnapshots:int, ingestion_date:datetime ) [
    1, true, 2, datetime("2024-04-26"),
    2, false, 0, datetime("2024-04-26"),
    3, true, 0,datetime("2024-04-26"),
    4, true, 1,datetime("2024-04-26"),
    1, true, 0,datetime("2024-04-27"),
    2, true, 0,datetime("2024-04-27"),
    3, true, 0,datetime("2024-04-27"),
    4, true, 1,datetime("2024-04-27")
    ];
    T
    | order by DiskId asc, ingestion_date asc 
    | extend new_setting = tostring(bag_pack("IsLeased", tostring(IsLeased), "NumSnapshots", tostring(NumSnapshots))) 
    | project-away IsLeased, NumSnapshots
    | extend old_setting=prev(new_setting)
    | where prev (new_setting) != new_setting and DiskId == prev(DiskId)
    

    This would give you a view at least on the changed rows:

    enter image description here