springspring-bootlogginglogbackgrafana-loki

delay in receiving loki logs from spring application


I created a simple spring boot application and added a loki4j dependency to it to send logs to loki. Logs are displayed in loki, but this happens with a significant delay (on average 30 seconds). Moreover, I noticed that if you do load testing, the logs will be updated faster, whereas when sending a single request, you have to wait a long time for it to appear in loki. Also, if you stop the application, all the logs appear at once. What could be the reason for such strange behavior? Here is my docker-compose file:

services:
  grafana:
    build: './config/grafana'
    ports:
      - "3000:3000"
    volumes:
      - ./grafana:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    networks:
      monitoring:
        aliases:
            - grafana
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    user: "root"
    volumes:
      - ./loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      monitoring:
        aliases:
          - loki

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus:/prometheus
    networks:
      monitoring:
        aliases:
            - prometheus
networks:
  monitoring:

Logback configuration:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <springProperty scope="context" name="name" source="spring.application.name" />

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>
                %d{HH:mm:ss.SSS} %highlight(%-5level) [%cyan(%thread)] %green(%logger{36}) - %msg%n
            </pattern>
        </encoder>
    </appender>

    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http>
            <url>http://192.168.0.114:3100/loki/api/v1/push</url>
        </http>

        <format>
            <label>
                <pattern>app=${name},host=${HOSTNAME},level=%level</pattern>
                <readMarkers>true</readMarkers>
            </label>
            <message>
                <pattern>
                    {
                    "level":"%level",
                    "class":"%logger{36}",
                    "thread":"%thread",
                    "message": "%message",
                    "requestId": "%X{X-Request-ID}"
                    }
                </pattern>
            </message>
        </format>
    </appender>

    <logger name="ru.alex" level="DEBUG" additivity="false">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="LOKI" />
    </logger>

    <root level="INFO">
        <appender-ref ref="LOKI"/>
        <appender-ref ref="CONSOLE"/>
    </root>
</configuration>

Solution

  • You can use the batchTimeoutMs configuration to set the time in which you want the appender to send the logs. The default value for it is 60000ms

    From the documentation: loki4j.github.io/loki-logback-appender/docs/configuration

    (added answer as the suggestion in comments helped OP)