bash docker docker-compose ansible monit

ansible succeeds but doesn't start flocked, piped, detached docker compose via bash script

Expected behaviour

ansible starts docker compose with flock and logging to file using the bash script without delay.

Observed behaviours

service does not get launched via ansible. Monit starts service correctly via bash script but delayed.

Description

I run updates via ansible and want to restart/rebuild docker-compose with the bash script via ansible without delay if possible.

Bash script

#! /bin/bash

lockfile="/var/run/lock/captain-hook"
export ENVIRONMENT=production

cd /home/captain-hook/captain-hook
flock -n "$lockfile" -c "docker-compose up --build --force-recreate >> /var/log/captain-hook/log 2>&1 &"
exit 0

Ansible task file

- name: stop container
  command: docker stop captain_hook

- name: Sleep for 10 seconds
  wait_for:
   timeout: 10

- name: start docker-compose
  shell: |
    /etc/monit/scripts/start_captain_hook.sh
  args:
    chdir: /home/captain-hook/captain-hook

debug output from ansible

TASK [captain-hook : start docker-compose] *****************************************************************************************************
task path: /home/axel/Dropbox/0_Programming/trading/trade-cloud-ansible/roles/captain-hook/tasks/code-update.yml:24
<168.119.100.44> ESTABLISH SSH CONNECTION FOR USER: root
<168.119.100.44> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/%h-%r 168.119.100.44 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<168.119.100.44> (0, b'/root\n', b'')
<168.119.100.44> ESTABLISH SSH CONNECTION FOR USER: root
<168.119.100.44> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/%h-%r 168.119.100.44 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1663325556.880422-52394-195122740762188 `" && echo ansible-tmp-1663325556.880422-52394-195122740762188="` echo /root/.ansible/tmp/ansible-tmp-1663325556.880422-52394-195122740762188 `" ) && sleep 0'"'"''
<168.119.100.44> (0, b'ansible-tmp-1663325556.880422-52394-195122740762188=/root/.ansible/tmp/ansible-tmp-1663325556.880422-52394-195122740762188\n', b'')
Using module file /usr/lib/python3/dist-packages/ansible/modules/command.py
<168.119.100.44> PUT /home/axel/.ansible/tmp/ansible-local-52322ijhvkvis/tmplnis9k7e TO /root/.ansible/tmp/ansible-tmp-1663325556.880422-52394-195122740762188/AnsiballZ_command.py
<168.119.100.44> SSH: EXEC sshpass -d10 sftp -o BatchMode=no -b - -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/%h-%r '[168.119.100.44]'
<168.119.100.44> (0, b'sftp> put /home/axel/.ansible/tmp/ansible-local-52322ijhvkvis/tmplnis9k7e /root/.ansible/tmp/ansible-tmp-1663325556.880422-52394-195122740762188/AnsiballZ_command.py\n', b'')
<168.119.100.44> ESTABLISH SSH CONNECTION FOR USER: root
<168.119.100.44> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/%h-%r 168.119.100.44 '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1663325556.880422-52394-195122740762188/ /root/.ansible/tmp/ansible-tmp-1663325556.880422-52394-195122740762188/AnsiballZ_command.py && sleep 0'"'"''
<168.119.100.44> (0, b'', b'')
<168.119.100.44> ESTABLISH SSH CONNECTION FOR USER: root
<168.119.100.44> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/%h-%r -tt 168.119.100.44 '/bin/sh -c '"'"'/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1663325556.880422-52394-195122740762188/AnsiballZ_command.py && sleep 0'"'"''
<168.119.100.44> (0, b'\r\n{"cmd": "/etc/monit/scripts/start_captain_hook.sh\\n", "stdout": "", "stderr": "", "rc": 0, "start": "2022-09-16 10:52:37.660030", "end": "2022-09-16 10:52:37.669701", "delta": "0:00:00.009671", "changed": true, "invocation": {"module_args": {"chdir": "/home/captain-hook/captain-hook", "_raw_params": "/etc/monit/scripts/start_captain_hook.sh\\n", "_uses_shell": true, "warn": true, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "executable": null, "creates": null, "removes": null, "stdin": null}}}\r\n', b'Shared connection to 168.119.100.44 closed.\r\n')
<168.119.100.44> ESTABLISH SSH CONNECTION FOR USER: root
<168.119.100.44> SSH: EXEC sshpass -d10 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/%h-%r 168.119.100.44 '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1663325556.880422-52394-195122740762188/ > /dev/null 2>&1 && sleep 0'"'"''
<168.119.100.44> (0, b'', b'')
changed: [trade-cloud] => {
    "changed": true,
    "cmd": "/etc/monit/scripts/start_captain_hook.sh\n",
    "delta": "0:00:00.009671",
    "end": "2022-09-16 10:52:37.669701",
    "invocation": {
        "module_args": {
            "_raw_params": "/etc/monit/scripts/start_captain_hook.sh\n",
            "_uses_shell": true,
            "argv": null,
            "chdir": "/home/captain-hook/captain-hook",
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": true
        }
    },
    "rc": 0,
    "start": "2022-09-16 10:52:37.660030",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}

TASK [captain-hook : report result] ************************************************************************************************************
task path: /home/axel/Dropbox/0_Programming/trading/trade-cloud-ansible/roles/captain-hook/tasks/code-update.yml:31
ok: [trade-cloud] => {
    "shell_result.stdout_lines": []
}

monit file to launch start script

# {{ ansible_managed }}

#==================== check start-captain-hook is running =======================
CHECK PROGRAM captainHook WITH PATH /etc/monit/scripts/check_captain_hook.sh
    START PROGRAM = "/etc/monit/scripts/start_captain_hook.sh"
    IF status != 0 FOR 1 CYCLES THEN START

Is there a way to make this work?

Solution

Thanks https://stackoverflow.com/users/2123530/%ce%b2-%ce%b5%ce%b7%ce%bf%ce%b9%cf%84-%ce%b2%ce%b5 and https://stackoverflow.com/users/9401096/zeitounator for your input!

I do now run docker compose detached and log to journald instead of piping to log file.

bash script

#! /bin/bash

lockfile="/var/run/lock/captain-hook"
export ENVIRONMENT=production

cd /home/captain-hook/captain-hook
flock -n "$lockfile" -c "docker-compose up --build --force-recreate -d"
exit 0

Logging to journald via this answer: Access logs of a killed docker container and documentation: https://docs.docker.com/config/containers/logging/configure/

/etc/docker/daemon.json

{
    "log-driver": "journald"
}

Changed behavior

Journald will only record the docker logs, therefore I am losing information how the container is build when monit starts the service. When ansible uses the bash script I now get the build info the ansible machine (my laptop^^).

Ansible does not flock successfully via the bash script but thats ok.

Monit also checks if the container is up via docker top captain_hook. If that ever fails (which it has in the past) then monit will correctly use flock.

The worst thing that can happen is that docker top captain_hook fails. The container would get killed once by restart via monit. I can live with that.