pythonloggingprometheusgrafanadata-pipeline

Can I use Prometheus to list the files processing or already processed?


I need to know the time per service of an application, which is processing some files. So I mean: the same file passes through each service and I need to know each pipeline time. Is that possible with Prometheus and, for example, Grafana? Or there is another tool for it? Or even... do I need to implement it on my own? (Obs: the services run in Python)


Solution

  • From what I understand from the question, what is being looked for is custom metrics. Prometheus is widely used for gathering metrics. You can use a library like prometheus_client and instrument the time taken to process the files in each stage.

    If the services that process the files are not batch jobs or cronjobs and can expose API endpoints, expose the metrics on, for example, "/metrics". This is only the publishing part. The metrics endpoint can then be consumed by Prometheus service using its scrape_config configurations. Read more about it here.

    If the services cannot expose endpoints and hence metrics, they can "push" the metrics to a Prometheus Push Gateway, and Prometheus can be configured to scrape the gateway. Read more about it here.

    It also has to be noted that it will not be advisable to try and publish metrics per file. The general practice is to publish metrics per file type.

    Once all the metrics are available in Prometheus, Grafana can then read from Prometheus and display graphs.

    There are a myriad of other architectural decisions one may need to take while setting it all up.