I am using ADF to parse a series of Avro files. The tricky part about this exercise is that the payload will contain a metrics array of varying sizes. And within each metric, there will be slightly different schemas.
This is what I have so far. I have flattened down the file so that I have this column.
So I will have an array of "payload metrics"....
..... and as you can see, there might be one or more levels of nesting at this level.
Long story short, what is the best way to proceed if I want to continue to break up this json node? Is there a way to dynamically break out the columns in the payload?
+Add mapping
-> Rule based mapping
true()
and check theDeep column traversal
checkbox.This will dynamically flatten all the subcolumns of complex field.
Reference: https://learn.microsoft.com/en-us/azure/data-factory/data-flow-flatten#rule-based-mapping