I have installed loki and grafana via helm charts to my cloud hosted k8s cluster. How do I send the loki metrics to grafana cloud? I know I should edit the promtail config file but how do I locate and view/edit helm chart files?
My installation procedure for loki+grafana are:
helm upgrade --install loki --namespace=monitoring grafana/loki-stack
helm upgrade --install grafana --namespace=monitoring grafana/grafana
kubectl port-forward service/grafana 3000:80 -n monitoring
This installation of loki only exposes loki to grafana locally. I wish to enter the configuration of loki (or more specifically promtail) so that I can send the loki logs to my grafana cloud account and monitor the logs from grafana cloud as well. I have done this in a non-k8s setting by modifying the promtail-config.yaml which is referenced to send the logs to grafana cloud. Adding the grafana cloud url to the promtail config such as:
clients:
- url: http://loki:3100/loki/api/v1/push
- url: https://123456:ekw0...=@logs-prod3.grafana.net/api/prom/push
When I explore grafana cloud > intergrations and connections > loki hosted logs > k8s cluster. I receive the following instructions
Your configuration, complete with your API key, has been generated below. Copy and paste this code to promtail/config.yaml to send data using promtail.
The code snippet:
curl -fsS https://raw.githubusercontent.com/grafana/loki/master/tools/promtail.sh | sh -s 123456 eB9... logs-prod3.grafana.net default | kubectl apply --namespace=default -f -
I am new to helm charts, so I do not know how to view the helm config files and where they are stored. Or how to obtain the promtail/config.yaml file from helm or the k8s cluster.
I'm not an expert in Loki but I downloaded the default values.yaml that Chart uses and there seems an option to mention the Promtail configuration which should help you (shown below) and looking at the commands above, it seems Helm would have used default values to install the chart, you can still modify them in 2 ways.
1st: Run the below to get the values.yaml that was used to install the chart by default, modify the Promtail configuration and then upgrade the installation using the modified values.yaml
helm get values -n monitoring grafana/loki-stack loki > values.installed.loki.yaml
*Modify the values.installed.loki.yaml, on line number 5 there are a few *promtail configuration you may need to set
promtail:
enabled: true
config:
lokiAddress: http://{{ .Release.Name }}:3100/loki/api/v1/push
Once done, upgrade the chart with the modified values with -f flag
helm upgrade --install loki --namespace=monitoring grafana/loki-stack -f values.installed.loki.yaml
2nd: Get the default values.yaml as shown below which is used for Loki installation, modify the promtail configuration and upgrade the installation as shown below
helm show values grafana/loki-stack > values.loki.yaml
Modify values.loki.yaml for the promtail configuration and upgrade the chart
promtail:
enabled: true
config:
lokiAddress: http://{{ .Release.Name }}:3100/loki/api/v1/push
Run an upgrade on the existing installation with the new values.yaml
helm upgrade --install loki --namespace=monitoring grafana/loki-stack -f values.loki.yaml