AWS healthd daemon warning - nginx log file does not exists

I have deployed a nodejs app using elasticbeanstalk with nginx.

When I check the log directory /var/log/nginx/

healthd folder exists and
nginx is creating /var/log/nginx/healthd/application.log.2019-10-04-13

But in /var/log/healthd folder there is a daemon.log file in which I am getting below error after every 5 seconds.

    # Logfile created on 2019-10-04 13:46:46 +0000 by logger.rb/47272
A, [2019-10-04T13:46:46.849261 #8094]   ANY -- : healthd daemon 1.0.3 initialized
W, [2019-10-04T13:46:47.011762 #8094]  WARN -- : log file "/var/log/nginx/healthd/application.log.2019-10-04-13" does not exist
W, [2019-10-04T13:46:52.012037 #8094]  WARN -- : log file "/var/log/nginx/healthd/application.log.2019-10-04-13" does not exist
W, [2019-10-04T13:46:57.012270 #8094]  WARN -- : log file "/var/log/nginx/healthd/application.log.2019-10-04-13" does not exist
W, [2019-10-04T13:47:02.012593 #8094]  WARN -- : log file "/var/log/nginx/healthd/application.log.2019-10-04-13" does not exist
W, [2019-10-04T13:47:07.012811 #8094]  WARN -- : log file "/var/log/nginx/healthd/application.log.2019-10-04-13" does not exist
W, [2019-10-04T13:47:12.013037 #8094]  WARN -- : log file "/var/log/nginx/healthd/application.log.2019-10-04-13" does not exist
W, [2019-10-04T13:47:17.013245 #8094]  WARN -- : log file "/var/log/nginx/healthd/application.log.2019-10-04-13" does not exist
W, [2019-10-04T13:47:22.013467 #8094]  WARN -- : log file "/var/log/nginx/healthd/application.log.2019-10-04-13" does not exist
W, [2019-10-04T13:47:27.013679 #8094]  WARN -- : log file "/var/log/nginx/healthd/application.log.2019-10-04-13" does not exist
W, [2019-10-04T13:47:32.013903 #8094]  WARN -- : log file "/var/log/nginx/healthd/application.log.2019-10-04-13" does not exist

So how to make daemon read nginx healthd log files?

Solution

This is an old post, but in case anyone is still dealing with this issue, here’s how I resolved it when I encountered the same problem.

If you have enhanced monitoring enabled, the access log configuration in your nginx.conf shoulld look something like this. Meaning nginx access logs get rotated every hour.

access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour   healthd;

However, nginx doesn’t create the hourly log file until it actually needs it — specifically, right before the first request of the new hour is logged.

But with enhanced monitoring enabled, healthd will attempt to read the nginx log file every 5 seconds. The warnings I saw were due to healthd not finding the log file because nginx hadn’t created it yet, due to a lack of requests.

I fixed the warnings by

Creating a cron that runs 5 minutes before the hour and preemptively create an empty nginx log file for the next hour.
Configuring healthd for enhanced health monitoring following the same article @ray mentions in his answer. The multi-container Docker sample specifically.
Lastly, overwrote the default ElasticBeanstalk healthd cleanup cron to clean up logs older than a day rather than an hour. This one is probably not necessary, but I implemented it while I was still thinking it was a timezone issue. I keep it now as a safety measure.

In ebextensions, it looks something like this:

files:
  /usr/local/sbin/setup-healthd-appstat.sh:
    mode: '000755'
    owner: root
    group: root
    content: |
      #!/bin/sh
      set -e
      mkdir -p /var/log/containers/nginx-proxy/healthd
      chmod 777 /var/log/containers/nginx-proxy/healthd
      if [ -d "/etc/healthd" ]
      then
        echo >> /etc/healthd/config.yaml
        echo "appstat_log_path: /var/log/containers/nginx-proxy/healthd/application.log" >> /etc/healthd/config.yaml
        echo "appstat_unit: sec" >> /etc/healthd/config.yaml
        echo "appstat_timestamp_on: completion" >> /etc/healthd/config.yaml
        systemctl start healthd || systemctl restart healthd
      fi
  /usr/local/sbin/create-next-healthd-log-file.sh:
    mode: '0755'
    owner: root
    group: root
    content: |
      #!/bin/bash
      NGINX_LOG_DIR="/var/log/nginx/healthd"
      NEXT_HOUR=$(date -d "+1 hour" +"%Y-%m-%d-%H")
      NEXT_LOG_FILE="$NGINX_LOG_DIR/application.log.$NEXT_HOUR"
      mkdir -p $NGINX_LOG_DIR
      if [ ! -f "$NEXT_LOG_FILE" ]; then
        touch "$NEXT_LOG_FILE"
        chown nginx:nginx "$NEXT_LOG_FILE"
        chmod 644 "$NEXT_LOG_FILE"
        echo "Created empty log file for next hour: $NEXT_LOG_FILE"
      fi
  /etc/cron.d/create_next_healthd_log_file:
    mode: '0644'
    owner: root
    group: root
    content: |
      55 * * * * root /usr/local/sbin/create-next-healthd-log-file.sh

container_commands:
    # Clean healthd files after a day rather than after an hour
    01_override_healthd_cleanup:
      command: |
        echo '#!/bin/sh
        find /var/log/nginx/healthd -type f -mtime +1 | grep -v application.log.$(date -u +"%Y-%m-%d-%H") | xargs rm -f' > /etc/cron.hourly/cron.logcleanup.elasticbeanstalk.healthd-proxy.conf
        chmod 755 /etc/cron.hourly/cron.logcleanup.elasticbeanstalk.healthd-proxy.conf
    02_configure_healthd:
        command: /usr/local/sbin/setup-healthd-appstat.sh
    # Proactive create to avoid healthd monitoring "file not found" warning before any requests have been made
    03_create_next_healthd_log_file:
        command: /bin/bash /usr/local/sbin/create-next-healthd-log-file.sh