I'm unable to shut down Docker containers that were launched by supervisor through supervisorctl stop all. Even through supervisorctl status shows the containers are down, docker ps and ps indicate that they are in fact still running.
Consulting the supervisor documentation on the action for supervisorctl stop <name> reveals that SIGTERM is sent to processes followed by SIGKILL if still running after some grace period. I tried to do this manually and found that
SIGTERM sent to a docker run process doesn't do anythingSIGKILL does kill the process, but doesn't actually update docker. docker ps shows this container is still runningSIGKILL doesn't shut down the containerThe questions is: How do I properly shut down a Docker container by supervisor?
Here's the result of my experiments simulating supervisor:
Starting position: foo-1 and bar-1 are running (I left the GCE containers in, in case they make a difference). ps aux and docker ps are in sync.
me@devenv:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5ba70bf8937f me/app:foo "/bin/sh -c 'supervi 5 minutes ago Up 5 minutes foo-1
e1a684bcfceb me/app:bar "/bin/sh -c 'supervi 5 minutes ago Up 5 minutes bar-1
fce5db0517df google/cadvisor:0.8.0 "/usr/bin/cadvisor" 35 minutes ago Up 35 minutes bbbb
db677eed47ef kubernetes/pause:go "/pause" 35 minutes ago Up 35 minutes 0.0.0.0:4194->8080/tcp aaaa
me@devenv:~$ ps aux | grep "docker run"
root 23358 0.0 0.1 124092 11856 pts/0 Sl 02:05 0:00 docker run --rm --name foo-1 ... -i me/app:foo
root 23365 0.0 0.1 124092 11928 pts/0 Sl 02:05 0:00 docker run --rm --name bar-1 ... -i me/app:bar
Simulate supervisorctl stop foo-1 by sending SIGTERM to the process. Result: process still active.
me@devenv:~$ sudo kill -SIGTERM 23358
... <waiting> ...
me@devenv:~$ ps aux | grep "docker run"
root 23358 0.0 0.1 124092 11856 pts/0 Sl 02:05 0:00 docker run --rm --name foo-1 ... -i me/app:foo
root 23365 0.0 0.1 124092 11928 pts/0 Sl 02:05 0:00 docker run --rm --name bar-1 ... -i me/app:bar
me@devenv:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5ba70bf8937f me/app:foo "/bin/sh -c 'supervi 6 minutes ago Up 6 minutes foo-1
e1a684bcfceb me/app:bar "/bin/sh -c 'supervi 6 minutes ago Up 6 minutes bar-1
fce5db0517df google/cadvisor:0.8.0 "/usr/bin/cadvisor" 36 minutes ago Up 36 minutes bbbb
db677eed47ef kubernetes/pause:go "/pause" 36 minutes ago Up 36 minutes 0.0.0.0:4194->8080/tcp aaaa
Next thing supervisor would do is issuing SIGKILL. Result: Process is killed (ps aux), but still showing as running docker process (docker ps).
me@devenv:~$ sudo kill -SIGKILL 23358
me@devenv:~$ ps aux | grep "docker run"
root 23365 0.0 0.1 124092 11928 pts/0 Sl 02:05 0:00 docker run --rm --name bar-1 ... -i me/app:bar
me@devenv:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5ba70bf8937f me/app:foo "/bin/sh -c 'supervi 19 minutes ago Up 19 minutes foo-1
e1a684bcfceb me/app:bar "/bin/sh -c 'supervi 19 minutes ago Up 19 minutes bar-1
fce5db0517df google/cadvisor:0.8.0 "/usr/bin/cadvisor" 49 minutes ago Up 49 minutes bbbb
db677eed47ef kubernetes/pause:go "/pause" 49 minutes ago Up 49 minutes 0.0.0.0:4194->8080/tcp aaaa
Supervisor was shut down during the above experiment (to avoid its autostart behaviour to interfere). The result for explicitly sending SIGKILL to the process could not be achieved by supervisor; the process was still alive (even though the supervisor logs state otherwise).
docker stop <container_id> did however stop the container.
Update
Inside the Docker containers is also a supervisord process running which manages certain processes. Perhaps the problem is that signals aren't propagated and thus it won't shut down...
Update 2
I narrowed down the problem. I was able to directly start a containers process from the Dockerfile instead of going through starting supervisord and it makes a difference. I'm able to control this container through supervisor (the one outside of the docker containers which controls the containers).
Update 3
Setting stopasgroup=true as suggested here does not change anything for me.
Update 4
I was able to solve one of the problems: supervisorctl not being able to shut down the process. The problem was that I was launching the docker containers in the supervisor configuration file with command=sudo docker run... which created a sudo docker run... and a docker run... process. supervisorctl stop... just terminated the sudo docker run... process while the actual docker process was still running. When I omit the sudo command, only 1 process per supervisor program is started and supervisorctl stop terminates the process.
One problem remains which is that docker ps shows the container is still running while ps aux doesn't. Oddly enough the containers still seem to be active as they respond to requests. A quick look at the process list confirms that all processes spawned by the docker container are still active, yet the docker run... process is missing from the process list.
Update 5
Sending SIGTERM, SIGHUP or SIGQUIT to the docker run process doesn't seem to do anything to the process. Only SIGKILL terminates the docker process properly. Supervisor gets updated properly, but docker ps still shows the docker process running.
I think I found the problem. I didn't realise it, but there's multiple ways of starting a program when a docker container is fired up.
Apparently CMD myexec param1 param2 starts a shell which in turn starts myexec (in fact these two processes are visible in the container with /bin/sh -c myexec... at PID 1. A better way is to start the program directly (in my case supervisord).
On the other hand, CMD ["/usr/bin/python", "/usr/local/bin/supervisord", "-c", "/root/supervisord.conf", "--nodaemon"] worked fine. I am now able to start and stop the docker container through supervisor.
Here's the relevant section in the docker docs:
The CMD instruction has three forms:
CMD ["executable","param1","param2"](exec form, this is the preferred form)
CMD ["param1","param2"](as default parameters to ENTRYPOINT)
CMD command param1 param2(shell form)
Update
Example supervisor file (inside Docker container):
[program:app]
command=python run_web_server.py
stdout_logfile=/var/log/app/app.log
directory=/opt/app
autostart=true
autorestart=false
stopsignal=INT
redirect_stderr=true
startretries=0
stopasgroup=true
killasgroup=true
[unix_http_server]
file=/var/run/supervisor.sock
chmod=0700
[supervisord]
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
childlogdir=/var/log/supervisor
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock
The mako template to generate the Docker (outside) supervisor file:
[program:container]
command=docker run --rm --name ${name} \
% if container_links is not UNDEFINED:
% for host in container_hosts:
--add-host ${host['name']}:${host['ip']} \
% endfor
% endif
% if container_links is not UNDEFINED:
% for link in container_links:
--link ${link}:${link} \
% endfor
% endif
% if port_mappings is not UNDEFINED:
% for ext in port_mappings:
-p ${ext}:${port_mappings[ext]} \
% endfor
% endif
-e "INSTANCE_NAME=${name}" \
-e "TZ=${timezone}" \
% if environ is not UNDEFINED:
% for k in environ:
-e "${k}=${environ[k]}" \
% endfor
% endif
-v ${deployment_dir}/tmp:${deployment_dir}/app/tmp \
... more -v
-i foo/app-${version}:${type}
stdout_logfile=${deployment_dir}/log/${name}.log
redirect_stderr=true
autostart=false
autorestart=false
% if priority is not UNDEFINED:
priority=${priority}
% endif
startretries=0
# stopasgroup=true
# killasgroup=true