sshgitlabsubprocesspipeline

How to tell GitLab not to wait for sub process when remotly running sh script?

I use a GitLab pipeline to

a) stop a python process on a remote server

b) deploy a new version of python code

c) trigger the restart of a python process (and do not wait for it to finish)

I have issues with the last step c).

The job for a) works without issues and immediately finishes. Therefore, there does not seem to be a general issue with remote execution of *.sh scripts using ssh in GitLab pipelines.

The job for c) does start the python process and the helper script that I use for it seems to have been finished. However, the GitLab job is stuck and keeps waiting after running the remote script.

Here is the ssh command used for job c) in my gitlab-ci.yml:

ssh -p 222 user@project.my-server.de ./path/to/start_server.sh

Content of the remote script start_server.sh:

#!/bin/bash
#set -xv

cd my_directory
# Installs python dependencies
# The grep part removes "already satisfied" messages from the pip3 install output
pip3 install -r requirements.txt  | grep -v 'already satisfied'
cd src
(
tmux -c "python3 main.py > /dev/null" # starts server in background task to keep it open after script finished
) &

At the end of the output of the GitLab job I see three dots ... denoting, that the job keeps running:

enter image description here

=> How can I correctly tell GitLab and/or ssh to

I tried several approaches that did not help:

a) Add & at end of line:

ssh -p 222 user@project.my-server.de ./path/to/start_server.sh &

b) Use nohup command:

nohup ssh -p 222 user@project.my-server.de ./path/to/start_server.sh

or

ssh -p 222 user@project.my-server.de nohup ./path/to/start_server.sh

or in start_server.sh:

nohup python3 main.py &

c) Include extra bash command :

ssh -p 222 user@project.my-server.de bash ./path/to/start_server.sh

d) Use -t option:

ssh -p 222 -t user@project.my-server.de ./path/to/start_server.sh

e) Update gitlab-runner to current version 15.9.1

f) Enable Gitlab debug trace

job_back_start:
  variables:
    CI_DEBUG_TRACE: "true"

Related:

https://forum.gitlab.com/t/gitlab-com-runner-never-finishes-after-script-is-completed/63817

https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4025

https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1979

https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1962

https://serverfault.com/questions/962452/how-to-use-nohup-properly-within-an-ssh-session-invoked-by-gitlab-runner

Edit:

If I include the verbose option -v the job finishes sometimes. However, it still takes > 1 minute, mostly waiting after the script has already been executed:

ssh -p 222 -v user@project.my-server.de ./path/to/start_server.sh

enter image description here

...

enter image description here

The waiting step, that I would like to avoid, takes place between lines 107 and 108 of the output, after I already got the return value:

enter image description here

Solution

  • a) I got it working using the at command:

    ssh -p 222 user@project.my-server.de "at now < ./path/to/start_server.sh"

    Also see related question

    https://serverfault.com/questions/962452/how-to-use-nohup-properly-within-an-ssh-session-invoked-by-gitlab-runner

    b) What also worked was to wait a few seconds and kill the ssh process:

     - ssh -p 222 user@project.my-server.de ./path/to/start_server.sh &
 - sleep 5
 - pkill ssh

    c) The content of the start_server.sh script can be simplified to

    #!/bin/bash
#set -xv

cd my_directory
# Installs python dependencies
# The grep part removes "already satisfied" messages from the pip3 install output
pip3 install -r requirements.txt  | grep -v 'already satisfied'
cd src
python3 main.py