Ingest Yahoo Finance data with Azure Data Factory
I'm trying to ingest data from the open-source, public Yahoo Finance API using Azure Data Factory. The endpoint I'm testing is https://query2.finance.yahoo.com/v8/finance/chart/GOLD.
I am able to ingest the data but I'm coming up with an issue when trying to transform the data as part of a data flow. I am trying to flatten the JSON produced, which is a series of nested arrays in the structure of:
To produce a table in the below format:
| timestamp | volume | open | low | high | close |
|---|---|---|---|---|---|
The setup of my flatten activity is as follows:
The Partition option I'm using is Use current partitioning. This is what it looks like under the Inspect tab:
However, when I try to preview the data, nothing comes up and the notifications in ADF show this error:
Could not fetch statistics due to operation timeout.
In the source, I've tried sampling the data to only 10 rows and I'm getting the same error so I don't think this is the issue. I have also tried a different API endpoint (MSFT) and I'm getting the same error here as well.
Any ideas appreciated!
Thanks,
Carolina
Figured it out! It was because the amount of data it was trying to ingest was too large. I set the query parameters as below and I'm now getting data through:
- Search for a qualifying subarray in a 3d array
- Adding values to a C# array
- Search a 2d array by one column value and return another value from the qualifying row
- Search a multidimensional array by one column value and return another value from the qualifying row
- Get element value from a 2d array row which qualifies because of another column value
- Can I use array_search() to find a whole-row match in a 2d array?
- How to Mark Repeated Entries as True Starting from the Second Occurrence Using NumPy?
- Why arrays are still there when we are actually always using List<T>?
- php - find if an array contains an element
- Doctrine entity object to array
- Determine if an associative array of strings or arrays is equal to another array with the same structure
- Printing array contents in the same line in awk
- Merge two 2d arrays, group by one column and sum another column within each group
- PHP json_decode multiple arrays or array in array to database
- Return the first level key when a column value is found in a 2d array
- Find key of parent in array / PHP
- Find parent key of match in a multidimensional array
- Recursively search a multidimensional array for the parent key of a matched key or value
- Find a parent key in a 2d array
- Get first level key in a 2d array of the first row with a specified value in any element
- Filter a 2d array to only keep rows with a specified value in a specified column
- Access an accompanying value in the first qualifying row of a 2d array
- Get the first level key of a 2d array row with a qualifying value in a specified column
- Get the first level key of the first row containing a specified column value in a 2d array
- Remap values of a flat array based on row data in a 2d array
- Recursively search a multidimensional array for a row with a qualifying column value
- Return values of qualifying row when searching a 2d array
- Get all first level keys from a 2d array where a specified value is found anywhere in the row
- Get the first qualifying whole row from a 2d array
- Recursively search a hierarchical multidimensional array for partial value matches in a column and return values in another column found on the way