httpsgrafanaprometheus-alertmanagergrafana-lokipromtail

How to connect Grafana with Loki over HTTPS?

I am trying connecting Grafana with Loki over HTTPS. I have the certificates for both Grafana and Loki. They are signed by my organization, and they have provided me with the CA.cer and file.pfx(bundle containing private key, public key, certificate chain & key pair) also password protected.

What I've done so far is I've exported the private key from pfx as Grafana.pfx and certificate chain from pfx as grafana.cer file for Grafana and exactly the same from Loki certificates.

Below is the Grafana configuration for HTTPS

[server]
# Protocol (http, https, h2, socket)
protocol = https

# The ip address to bind to, empty will bind to all interfaces
http_addr = 0.0.0.0

# The http port to use
http_port = 3000

# The public facing domain name used to access grafana from a browser
domain = grafana.xyz.net

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
enforce_domain = false

# The full public facing url
root_url = %(protocol)s://%(domain)s:%(http_port)s/

# https certs & key file
cert_file = /opt/grafana/certs/grafana.xyz.net.cer
cert_key = /opt/grafana/certs/grafana.xyz.net.key

While below is the Loki configuration for HTTPS

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

  http_tls_config:
   client_auth_type: RequireAndVerifyClientCert    
   client_ca_file: /opt/loki/certs/loki_CA_.cer   
   cert_file: /opt/loki/certs/loki.xyz.net.cer
   key_file: /opt/loki/certs/loki.xyz.net.key
    
  grpc_tls_config:
   client_auth_type: RequireAndVerifyClientCert
   client_ca_file: /opt/loki/certs/loki_CA_.cer   
   cert_file: /opt/loki/certs/loki.xyz.net.cer
   key_file: /opt/loki/certs/loki.xyz.net.key

Well now, when I try to hit the URL of the Loki from the Grafana dashboard > connections > data sources and type the URL field as "https://loki.xyz.net:3100" I see an error as below,

Grafana logs on the console is as

logger=context userId=1 orgId=1 uname=admin t=2023-08-24T16:50:55.86218229+03:00 level=error msg="Failed to call resource" error="Get \"https://loki.xyz.net:3100/loki/api/v1/labels?start=1692884455845000000&end=1692885055845000000\": tls: failed to verify certificate: x509: certificate signed by unknown authority" traceID=

Loki logs on the console is as

TLS handshake error from 10.244.199.30:33762: remote error: tls: bad certificate

Been troublshooting this for long time and its not working whatever I try. Just to let you know that grafana is connecting perfectly fine over http.

What exactly am I missing? I couldnt find much content about this over the internet. If someone can pin point me that what all files shall I place on the config files for each application for them to communicate over https.

Solution

  • Well, the issue has been solved, it was quite a silly one. Not to mention, there was no where mention in the official documentation about this.

    Posting an answer as it can help someone.

    Enable TLS Client Auth

    You just literally have to enable the option "TLS Client Auth" from the Grafana dashboard and this option will read the configuration file of the Grafana where you have defined the key and the certificate.