Im trying to collect logs, that generate my services in docker containers using grafana/loki. I use Serilog in my asp.net application for sending logs to loki, which is selected as a data source for displaying in grafana. My application in debug mode in IDE sends logs, and graphana get them, but these same applications, working in a docker container, do not send logs to a container with loki.

Here is the composition, configuration and application code:

Docker-compose file:

version: "3.3"
networks:
  loki:
services:
  myservice.api:
    image: myservice.api
    build:
      context: ../
      dockerfile: ./Source/Services/myservice/myservice.api/Dockerfile
    container_name: myservice
    ports:
      - "${MYSERVICE_API_PORT}"
    environment:
      - ${MYSERVICE_API_URLS}
    networks:
      - loki
  loki:
      container_name: loki
      image: grafana/loki:2.4.0
      volumes:
        - ./Monitoring/config/loki:/etc/loki
      ports:
        - "3100:3100"
      restart: unless-stopped
      command: -config.file=/etc/loki/local-config.yaml
      networks:
        - loki

  grafana:
      container_name: grafana
      image: grafana/grafana:latest
      user: "1000"
      volumes:
        - ./Monitoring/config/grafana:/etc/lib/grafana
      ports:
        - "3000:3000"
      environment:
        - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
        - GF_AUTH_ANONYMOUS_ENABLED=true
        - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      entrypoint:
        - sh
        - -euc
        - |
          mkdir -p /etc/grafana/provisioning/datasources
          cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
          apiVersion: 1
          datasources:
          - name: Loki
            type: loki
            access: proxy
            orgId: 1
            url: http://loki:3100
            basicAuth: false
            isDefault: true
            version: 1
            editable: true
            apiVersion: 1
          EOF
          /run.sh
      restart: unless-stopped
      networks:
        - loki

Loki config:

auth_enabled: false

server:
    http_listen_address: 0.0.0.0
    http_listen_port: 3100
    grpc_listen_port: 9096

common: 
    path_prefix: /tmp/loki
    storage: 
        filesystem:
            chunks_directory: /tmp/loki/chunks
            rules_directory: /tmp/loki/rules
    replication_factor: 1
    ring: 
        instance_addr: 127.0.0.1
        kvstore:
            store: inmemory

schema_config:
    configs:
        - from: 2023-08-22
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
          index: 
            prefix: index_
            period: 24h

ruler: 
    alertmanager_url: http://localhost:9093

Part of my application: Setup.cs:


public class Setup
    {
        /// <summary>
        /// Configure logger
        /// </summary>
        /// <returns>Configured logger</returns>
        public static LoggerConfiguration GetLoggerConfiguration()
        {
            return new LoggerConfiguration()
                .WriteTo.Console(LogEventLevel.Debug)
                .WriteTo.File(
                    new JsonFormatter(),
                    "../logs/log-.json",
                    LogEventLevel.Fatal,
                    rollingInterval: RollingInterval.Day)
                .WriteTo.GrafanaLoki(
                    "http://host.docker.internal:3100",
                    new List<LokiLabel>()
                    {
                        new()
                        {
                            Key = "Service",
                            Value = System.IO.Path.GetFileName(Assembly.GetExecutingAssembly().Location).Replace(".dll", "")
                        }
                    });
        }

Program.cs:

public class Program
    {
        public static void Main(string[] args)
        {   
            Log.Logger = Setups.GetLoggerConfiguration().CreateLogger();
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args)
        {
            return Host.CreateDefaultBuilder(args)
                    .ConfigureLogging((context, logging) =>
                    {
                        logging.SetMinimumLevel(LogLevel.Critical);
                        logging.ClearProviders();
                    })
                    .UseSerilog()
                    .ConfigureAppConfiguration((hostingContext, builder) =>
                    {
                        var hostingEnvironment = hostingContext.HostingEnvironment;
                        builder
                            .AddJsonFile("appsettings.json", true, true)
                            .AddJsonFile($"appsettings.{hostingEnvironment.EnvironmentName}.json", true)
                            .AddJsonFile("appsettings.local.json", true)
                            .AddEnvironmentVariables();
                            
                        if (args == null)
                        {
                            return;
                        }

                        builder.AddCommandLine(args);
                    })
                    .ConfigureWebHostDefaults(webBuilder => { webBuilder.UseStartup<Startup>(); }
                    );
        }

Startup.cs:

public class Startup
    {
        private readonly IConfiguration _configuration;
        public Startup(IConfiguration configuration)
        {
            _configuration = configuration;
        }

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddControllers();
            services.AddInfrastructureData(_configuration);
            services.AddSwaggerGen(c =>
            {
                c.SwaggerDoc(AppsettingConstants.Version,
                    new OpenApiInfo
                    {
                        Title = AppsettingConstants.ApplicationTitle, Version = AppsettingConstants.Version
                    });
                c.IncludeXmlComments(GetXmlCommentsFilePath());
                c.OperationFilter<DefaultResponceFilter>();
            });
            services.AddSwaggerExamples();
            services.AddApplication();
            services.AddInfrastructureRepositories();
            services.AddLogging();
        }

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            app.UseMiddleware<ExceptionHandlingMiddleware>();
            app.UseSerilogRequestLogging();
            app.UseSwagger();
            app.UseSwaggerUI(c =>
            {
                c.SwaggerEndpoint($"/swagger/{AppsettingConstants.Version}/swagger.json",
                    AppsettingConstants.ApplicationTitle);
            });
            app.UseRouting();
            app.UseEndpoints(ConfigureEndpoints);
        }

        protected virtual void ConfigureEndpoints(IEndpointRouteBuilder endpoints)
        {
            endpoints.MapControllers();
        }

        private static string GetXmlCommentsFilePath()
        {
            return typeof(Startup).GetTypeInfo().Assembly.GetName().Name + FileExtensionConstant.Xml;
        }
    }

Method:

public async Task<CategoryDto> UpdateCategory(ushort id, string fullName, string filePath,
            CancellationToken cancellationToken)
        {
            _logger.LogInformation(
                $"Request from {HttpContext.Connection.RemoteIpAddress} in CategoryController, " +
                $"http-method: {HttpContext.Request.Method}, SpanId: {Activity.Current?.SpanId}, TraceId: {Activity.Current?.TraceId}");
            var category = await _myservice.UpdateCategory(id, fullName, filePath, cancellationToken);
            return category;
        }

In the presented configuration, it turns out to collect logs of a service that is running in the IDE, but it does not work to collect logs from the same service running in the same container network with grafana and loki.

I tried to collect logs using a promtail configured to send loki, but it collected logs only of containers that are in the container on the path var/lib/log/dpkg.log.

Here is promtail configs:

server:
  http_listen_port: 9080
  grpc_listen_port: 0
  
clients:
  - url: http://docker.host.internal:3100/loki/api/v1/push

positions:
  filename: /tmp/positions.yaml
  
target_config:
  sync_period: 10s
  
scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log

It seems I need to create .log file using my app inside docker? My app also creates .log file in file system, when it in debug, but there is no new folders with logs in docker container file system.

Issue with collecting logs from asp-net services in containers using containers with grafana and loki

Here is the composition, configuration and application code:

Here is promtail configs: