Jenkins Docker Container: Ansible fails to connect via SSH when run via Jenkins, connects fine if not ran via Jenkins. Failed ssh: muxserver_listen"

I'm running a Docker container of Jenkins with Ansible in the same image. Ansible is updating another Docker container on the same Docker network.

Useing an ssh key file.

Tested using root user and user jenkins

Both users tested. I can ssh from the Jenkins/Ansible container to the target container using the CLI. I can run the playbook on the CLI and it works, it's only when I run the playbook through Jenkins pipline it failes.

If I run the same command that Jenkins tries, (copied from the Jenkins console). The only change I made was the tmp ssh key to the real key. The temp key comes from Jenkins credentials. The playbook works fine.

--private-key /var/lib/jenkins/workspace/TestJs/ssh9809173197875714932.key Real Key --private-key /var/lib/jenkins/.ssh/jenkins_key

/usr/bin/ansible-playbook /var/lib/jenkins/workspace/TestJs/deploy/playbooks/deployToTarget.yml -i /var/lib/jenkins/workspace/TestJs/deploy/playbooks/inventory.inv --private-key /var/lib/jenkins/.ssh/jenkins_key -u jenkins

The above call on the Jenkins Docker cli works as expected.

To rule out the Jenkins temp key and validate the Jenkins key credentials, I wrote a simple pipline that connects to the host directly and runs a shell script, that connects fine and work as expected.

Pipeline Snippet

stages {
        stage('Deploy on Test') {
      steps {
        ansiblePlaybook( credentialsId: 'TEST_HOST_SSH_KEY',
                 disableHostKeyChecking: true,
                 installation: 'Ansible',
                 inventory: "${WORKSPACE}/deploy/playbooks/inventory.inv",
                 playbook: "${WORKSPACE}/deploy/playbooks/deployToBSI_SMA.yml",
                 colorized: true)
                
        } 
        }
    }

The console shows this error everytime. I admit I am struggling with this one.

PLAY [Deploy On Test] ***************************************************

TASK [Gathering Facts] *********************************************************
[1;31mfatal: [bsi_sma]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: muxserver_listen: link mux listener /var/lib/jenkins/.ansible/cp/119f480311.4lIaQB6LhOC2svmf => /var/lib/jenkins/.ansible/cp/119f480311: Bad file descriptor", "unreachable": true}[0m

PLAY RECAP *********************************************************************
[0;31mbsi_sma[0m                    : ok=0    changed=0    [1;31munreachable=1   [0m failed=0    skipped=0    rescued=0    ignored=0 

Any help is much appreciated; I have spent far too many days on this issue.

UPDATE: Originally, tested both users in the CLI, as in -u jenkins or -u root

Both work while I'm in the CLI as root

If I run su jenkins and try running the same command in the CLI, I get the same error.

So the issue appears to be who the command is run under; Jenkins is running as user Jenkins.

UPDATE fixed it: Jenkins was not picking up the Ansible configuration in the playbook

The CLI was of course

I added this to the Jenkins pipeline and it fixed it

environment {
    ANSIBLE_CONFIG = "${WORKSPACE}/deploy/playbooks/ansible.cfg"
    
}

Jenkins running /usr/bin/ansible-playbook, was not picking up the Ansible configuration inside the playbook directory.

The CLI was, of course picking up the configuration.

I added this to the Jenkins pipeline, and it fixed it.

environment {
    ANSIBLE_CONFIG = "${WORKSPACE}/deploy/playbooks/ansible.cfg"
}

I also added this to my Ansible config.

ssh_args= -o ControlMaster=no

Now, it works as expected.

However, I'm unsure why it does not work if a control_path is set.