Ingesting logs from Azure Blob Storage to Loki

Similar to a previous question related to reading logs from Azure Storage Accounts for Fluentd/Fluentbit, I am searching for a more direct way to read logs from Azure Storage Accounts into Loki.

When searching, I found Amazon S3 plugins for Fleuntd/bit and Logstash, nothing for Azure Storage Accounts. Only Sumo Logic seems to support streaming logs from azure storage accounts.

Here seems to be a possibility to read Azure logs into Loki from Azure Event Hubs

I have implemented Java Azure Functions that get triggered when a storage event fires on the containers of the Azure Storage Account. Then, I am going to process and push the logs from the log files (in Json) to Loki via its API, or perhaps use a Java client help me with the pushing of the log lines to Loki.

My question is if someone got a better idea, like having plug-ins similar to the ones provided for Amazon S3 from Fluentbit/Fluentd or Logstash. When possible, I want to avoid using Azure Functions for Java.

Also, would it be better (when using Loki) to get the logs sent to an Azure Event Hub rather than Storage Accounts? The cost factor is critical for me, therefore I first opted to using Storage Accounts and not event hubs.

steps to Ingesting logs from Azure Blob Storage to Loki

• create a Event Hubs Namespace and Event Hub In azure portal go the Azure Blob Storage and select Events on the left menu, and then select + Event Subscription on the toolbar.

• Enter a name for the event subscription.

• Enter a name for the system topic. A system topic provides an endpoint for the sender to send events. For more information, see System topics

• select the end point type as Event hub

• Event subscription triggers when actions, such as those involving blobs, file shares, queues, tables, and any action with their respective blobs, occur and sends them to the Event Hub.

Please refer to this link for Azure Event Hubs to Loki and link1 to read logs from Azure Storage Accounts into Loki.

Steps to connect even hub to loki:

• Go the event hub and next, decide on the authentication method you'll use: either OAuth or connection string. If using OAuth, make sure you have the required credentials set up. If using a connection string, obtain it from Azure.

• In your Loki configuration file, set up the loki.source.azure_event_hubs component. Specify the Event Hub's namespace, list the Event Hubs you want to consume, and define where the logs will be forwarded.

• Configure the authentication block with your chosen method and credentials. You can also customize other settings like group ID, relabeling rules, and whether to use incoming timestamps.

• Ensure the destination specified in forward_to is correctly configured to receive logs. This destination could be an instance of LogsReceiver.