My task is to configure Docker log monitoring for around 70 containers. Consolidating all logs into a single panel is not ideal, especially during stress tests and monitoring Docker applications. I want to be able to select a Docker container application in a panel and see the corresponding logs. I have achieved this, but the problem is that the container names are encrypted, and I need to decrypt them.
My question is: How can I decrypt the container names using the Loki data source and Promtail?
Additionally, I noticed that the Prometheus data source displays decrypted container names, but the Loki data source does not. Here are my configurations for both:
Also, I am running Loki and Promtail like normal application with configured systemd daemon. Loki:
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 5m
chunk_retain_period: 30s
schema_config:
configs:
- from: 2021-03-08
store: boltdb
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb:
directory: /tmp/loki/index
Promtail:
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: cadvisor
static_configs:
- targets: ['localhost:8080/containers/']
labels:
job: cadvisonr
__path__: /containers
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
- job_name: containers
static_configs:
- targets:
- localhost
labels:
job: containerlogs
__path__: /var/lib/docker/containers/*/*log
pipeline_stages:
- json:
expressions:
output: log
stream: stream
attrs:
- json:
expressions:
tag:
source: attrs
- regex:
expression: (?P<container_name>(?:[^|]*[^|]))
source: tag
- timestamp:
format: RFC3339Nano
source: time
- labels:
# tag:
stream:
container_name:
- output:
source: output
Also, I know about loki-driver-docker plugin, but I tried to run it and nothing changes for me, and documentation is terrible, it says just run it and almost nothing about configuration or how to check if it is working correct.
So, I would be glad to hear any ideas and suggestions. Forum is the last chance for me to solve this problem.
It works like this:
<something>_configs
static_config
only sees filesPromtail config may look like this:
- job_name: "docker"
docker_sd_configs:
- host: "unix:///var/run/docker.sock"
refresh_interval: "1s"
relabel_configs:
- source_labels: ['__meta_docker_container_name']
target_label: "container_name"
- source_labels: ['__meta_docker_container_id']
target_label: "container_id"
# etc.
Then in grafana-loki you can filter the logs using LogQL language, like:
logcli '{container_name="something"}'