loggingnginxprocessdocker

Logging from multiprocess docker containers


I'm using the nginx method of symlinking linking to /dev/stdout for any log files that I want to appear in 'docker logs', however this is not working.

I have tested this with a simple cronjob in /etc/crontab, if a symlink is present (pointing to /dev/stdout) it doesn't write anything (as far as I can tell), but if I delete the symlink and it writes to the file.

Also if I echo into /dev/stdout it is echo'd back on the command line however it isn't found in 'docker logs'...

Question: Should this work? (It seems to work with nginx). Else, how would I get logs from 'secondary' processes to appear in docker logs.

For ref:

Nginx Dockerfile showing the symlinking method: https://github.com/nginxinc/docker-nginx/blob/a8b6da8425c4a41a5dedb1fb52e429232a55ad41/Dockerfile

Created an official bug report for this: https://github.com/docker/docker/issues/19616

My Dockerfile:

FROM ubuntu:trusty
#FROM quay.io/letsencrypt/letsencrypt:latest # For testing

ENV v="Fri Jan 22 10:08:39 EST 2016"

# Setup the cronjob
ADD crontab /etc/crontab
RUN chmod 600 /etc/crontab

# Setup letsencrypt logs
RUN ln -sf /dev/stdout /var/log/letsencrypt.log
# Setup cron logs
RUN ln -sf /dev/stdout /var/log/cron.log
RUN ln -sf /dev/stdout /var/log/syslog

# Setup keepalive script
ADD keepalive.sh /usr/bin/keepalive.sh
RUN chmod +x /usr/bin/keepalive.sh

ENTRYPOINT /usr/bin/keepalive.sh

The crontab file:

* * * * * root date >> /var/log/letsencrypt.log

keepalive.sh script

#!/bin/bash

# Start cron
rsyslogd
cron

echo "Keepalive script running!"

while true; do

    echo 'Sleeping for an hour...'
    sleep 10

done

Solution

  • End result is that the /dev/stdout for the cron job was pointed to the different device.

    /proc/self/fd/1 and should have been /proc/1/fd/1 because as docker only expects one process to be running this is the only stdout it monitors.

    So once I had modified the symlinks to point at /proc/1/fd/1 it should have worked however apparmor (on the host) was actually denying the requests (and getting permissions errors when echoing to /proc/1/fd/1) because of the default docker profile (which is automatically generated but can be modified with --security-opts).

    Once over the apparmor hurdle it all works!

    This said, after looking at what is required to be modified in apparmor to allow the required request I decided to use the mkfifo method as show below.

    Dockerfile

    FROM ubuntu:latest
    
    ENV v="RAND-4123"
    
    # Run the wrapper script (to keep the container alive)
    ADD daemon.sh /usr/bin/daemon.sh
    RUN chmod +x /usr/bin/daemon.sh
    
    # Create the pseudo log file to point to stdout
    RUN mkfifo /var/log/stdout
    RUN mkfifo /var/log/stderr
    
    # Create a cronjob to echo into the logfile just created
    RUN echo '* * * * * root date 2>/var/log/stderr 1>/var/log/stdout' > /etc/crontab
    
    CMD "/usr/bin/daemon.sh"
    

    daemon.sh

    #!/bin/bash
    
    # Start cron
    cron
    
    tail -qf --follow=name --retry /var/log/stdout /var/log/stderr