Promtail Config And Setup For Docker Swarm Containers

I have a promtail and docker compose config and setup that works fine but when i try to follow same for docker swarm cluster, logs are not showing up for some reason

I have searched online for a doc on working config and setup for docker swarm with promtail and unfortunately i could not find anything to help

So here i am hoping someone will be willing to help point me in the right direction

here is error from promtail container in docker swarm

level=error ts=2024-11-21T00:52:59.569451224Z caller=client.go:430 component=client host=loki:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream"

level=error ts=2024-11-21T00:53:10.069885261Z caller=client.go:430 component=client host=loki:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream"

here is promtail config am using

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
    - targets:
        - localhost
      labels:
        job: varlogs
        # __path__: /var/log/*log
        __path__: /var/log/!(auth.log)*log

  - job_name: flog_scrape 
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: label
            values: ["logging=promtail"] 
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'logstream'
      - source_labels: ['__meta_docker_container_label_logging_jobname']
        target_label: 'job'
    pipeline_stages:
      - cri: {}
      - multiline:
          firstline: ^\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2},\d{3}
          max_wait_time: 3s
      # https://grafana.com/docs/loki/latest/clients/promtail/stages/json/
      - json:
          expressions:
            #message: message
            level: level
            #output: 'message'

and here is the docker swarm service i have the logging labels attached to

x-logging:
  &default-logging
  driver: "json-file"
  options:
    max-size: "1m"
    max-file: "1"
    tag: "{{.Name}}"

services:

  app-1:
    image: "app-1:v0.1"
    networks:
      - apps
    env_file:
      - ./env/.env.app-1
    deploy:
      mode: replicated
      replicas: 2
      placement:
        constraints:
          - node.labels.node != node-1
    labels:
      logging: "promtail"
      logging_jobname: "containerlogs"
    logging: *default-logging

networks:
  apps:
    external: true

but when i check loki, i do not see the container and logstream labels in loki dashboard like it does for docker compose one that works with the same above config

what am i doing wrong and what do i need to fix this so i can see the container and logstream lables to filter to the container logs i want to view logs for?

docker swarm labels are not showing up in loki

docker compose works fine and labels show up in loki

issue fixed, i needed to deploy promtail on docker swarm cluster as global mode, not on 1 node only

so updated promtail to be global deploy on all nodes in the cluster so it can scrpae the logs from the nodes

issue was i only had it on just one node and thus will not work to scrape logs for the other nodes

  promtail:
    image: grafana/promtail:2.9.1
    command: "-config.file=/mnt/config/promtail-config.yaml"
    volumes:
      - ./promtail/promtail-config.yaml:/mnt/config/promtail-config.yaml
      - /var/log:/var/log
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    deploy:
      mode: global