How to chuck JSON output efficiently to Azure Log Analytics workspace
I have data (650k rows) which i want to store using the Send data to Azure Log Analytics workspace module.
this is my workflow:
but i get this error:
"message": "The response is not in a JSON format.",
"innerError": "The page was not displayed because the request entity is too large."
when i send 20 rows worth of data it works.
I believe somewhere in the 650k rows a string breaks the json causing it to give me this error.
I believe Advanced Hunting is only giving me ~95k rows as an output, though 650k rows exist. maybe i have to structure it to give me 50k rows at a time and send that through.
whats the most efficient way to send this data in chucks using logic apps?
Here's a possible solution.
Gather what the max length of the data is:
Store this as a variable
Initialize a variable which is how many chunks we want. here I've split the total length by 10,000 . this is the number of chunks we wan.t
next we create an array of indexes, that is the length of the chunks i.e ([1,2,3,4,5,etc]).
length(variables('numberArray'))andadd(variables('lengthofarray'),1)
Now for every index in the array we send a chunk of data from the table. the between statement has
sub(mul(int(items('send_logs_to_LAW')),10000),10000)andmul(int(items('send_logs_to_LAW')),10000), this allows us to read only 10,000 rows and send 10,000 rows so we dont hit the limit on the send data module.
This is a solution that utilizes logic apps. However, there may be a better solution with DCR rules?
- Updating azure Storage Account table through Logic Apps
- Logic App AuthorizationFailure - vnet integration needed
- Use Salesforce Connectors from Hosted Azure Logic App in VSCode?
- Azure logic apps encode left bracket in json query param names to ~2. How can I rewrite with concat and/or replace functions?
- How to handle null value in logic app condition module
- Add properties to an object in a variable in Azure Logic Apps
- Workflow Validation Failed Error: The function cannot be called from a logic app. It must not have a custom route
- In an AZURE Function how to throw an error to be caught in the Azure Logic AP
- Cannot get AAD auth token in Logic Apps
- Parameterising connections.json in Azure Logic Apps(Standard) for deployment via Azure DevOps
- Logic Apps: How to use create a new watchlist with data (raw content) module
- Add extra attribute to object Logic app
- How to hide or change legend in "Run query and visualize results" logic app function?
- Azure Logic App Authorization Policy set via Bicep
- Azure Event Grid Graph API SharePoint Event info useful?
- Azure Logic App Function call for non json body
- Publish Failure Reporting Add or Build Definition File
- Azure Logic App returns metadata instead of SQL query results when invoked via HTTP trigger
- Azure Logic App is only inserting 100 rows into SQL Server table
- Unable to open the workflow in designer using vs code
- How to Pass SAS Token as Header in Azure Logic App HTTP Trigger?
- Azure Logic Apps with Form Recognizer - Get Array definition in optional settings entry
- logic app ( power automate) extract checklist items from a task from a planner plan
- Power Automate Get Filename if Contains Special Character
- Task AzureAppServiceSetting@1: Application Settings object is not a valid JSON while deploying with YAML file from Azure Devops
- unable to delete or move first step from logic app?
- Azure Logic App HTTP Request Authentication Scope
- Enabling minimum apiVersion to 2021-08-01 in Azure API Management causing saving issues or deployment errors for existing logic apps
- Azure Logic Apps: Set condition to False when SQL query returns no rows of data
- Send alert/Logic apps if Azure ML pipeline Completed with Azure ML Pipeline Outputs attached to email