kubernetesprometheusgrafanagrafana-lokikube-prometheus-stack

Loki that is set up as Prometheus datasource in order to have alerts on log counts, is wrongly configured, getting 404 error


We have added Loki twice as additional datasource for grafana in kube-prometheus-stack helm chart, the first is as type loki, the second one as type prometheus as the second is an only hack that allows grafana to have alerts from loki:

grafana:
  ## Configure additional grafana datasources (passed through tpl)
  ## ref: http://docs.grafana.org/administration/provisioning/#datasources
  additionalDataSources:
    - name: Loki
      type: loki
      access: proxy
      url: http://loki.monitoring.svc.cluster.local:3100
      jsonData:
        maxLines: 1000
    - name: LokiAsPrometheus
      type: prometheus
      access: proxy
      url: http://loki.monitoring.svc.cluster.local:3100
      jsonData:
        maxLines: 1000

But when I create a new dashboard and select LokiAsPromtheus as datasorce, grafana http responses respond with errors, like there is some misconfiguration.


Solution

  • It was critical to put /loki resource as the part of loki-as-prometheus url like that:

    grafana:
      ## Configure additional grafana datasources (passed through tpl)
      ## ref: http://docs.grafana.org/administration/provisioning/#datasources
      additionalDataSources:
        - name: Loki
          type: loki
          access: proxy
          url: http://loki.monitoring.svc.cluster.local:3100
          jsonData:
            maxLines: 1000
        - name: LokiAsPrometheus
          type: prometheus
          access: proxy
          url: http://loki.monitoring.svc.cluster.local:3100/loki
          jsonData:
            maxLines: 1000