Ansible Playbook fails on a particular server Only

Below ansible-playbook connectivity works fine for all my servers except dest_svr. Below command fails.

ansible-playbook -vvvv /opt/test/Auto/site.yml  -i /opt/test/Auto/hosts.txt -e "source_host=dest_svr execmd=echo source_user=user2"

(0, '', 'OpenSSH_7.7p1 (CentrifyDC build 5.5.1-395) , OpenSSL 1.0.2o-fips 27 Mar 2018\r\ndebug1: Reading configuration data /home/user1/.ssh/config\r\ndebug1: Reading configuration data /etc/centrifydc/ssh/ssh_config\r\ndebug1: /etc/centrifydc/ssh/ssh_config line 3: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 57768\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to dest_svr closed.\r\n') fatal: [dest_svr]: FAILED! => { "changed": false, "module_stderr": "OpenSSH_7.7p1 (CentrifyDC build 5.5.1-395) , OpenSSL 1.0.2o-fips 27 Mar 2018\r\ndebug1: Reading configuration data /home/user1/.ssh/config\r\ndebug1: Reading configuration data /etc/centrifydc/ssh/ssh_config\r\ndebug1: /etc/centrifydc/ssh/ssh_config line 3: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 57768\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 2\r\nShared connection to dest_svr closed.\r\n", "module_stdout": "/home/user2/.ansible/tmp/ansible-tmp-1549081965.65-147775141838271/AnsiballZ_setup.py: line 3: _ANSIBALLZ_WRAPPER: command not found\r\n/home/user2/.ansible/tmp/ansible-tmp-1549081965.65-147775141838271/AnsiballZ_setup.py: line 4: syntax error near unexpected token ('\r\n/home/user2/.ansible/tmp/ansible-tmp-1549081965.65-147775141838271/AnsiballZ_setup.py: line 4:def _ansiballz_main():'\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 2 } to retry, use: --limit @/opt/test/Auto/site.retry

PLAY RECAP ***************************************************************************************************************************************************************************************************************** dest_svr : ok=0 changed=0 unreachable=0 failed=1

Source Ansible Server details:

[user1@source_svr: Auto]$ uname -a
Linux source_svr 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[user1@source_svr: Auto]$ ansible --version
ansible 2.7.1
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/user1/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

Ansible Host file details:

cat /opt/test/Auto/hosts.txt
[dest_svr]
dest_svr ansible_ssh_common_args="-t -o UserKnownHostsFile=/dev/null  -o StrictHostKeyChecking=no -o ControlPersist=30m" ansible_shell_executable=/bin/bash ansible_shell_type=sh ANSIBLE_HOST_KEY_CHECKING=false ansible_python_interpreter= ansible_ssh_private_key_file=/opt/test/ssh_key/id_rsa USER_RUN=user2

And now the Destination Server details:

user2@dest_svr:~$ uname -a
SunOS dest_svr 5.11 11.3 sun4v sparc sun4v
user2@dest_svr:~$ python --version                                                                                                                                                                                    
Python 2.7.14

Note: I wish to make changes only to my playbook and do not wish to change anything in ansible.cfg as other playbooks should not be affected.

Solution

As best I can tell it's caused by this:

line 3: _ANSIBALLZ_WRAPPER: command not found

which is this line, but a little digging seems to imply that SunOS shebang lines do not tolerate pointing to an executable that is, itself, also a shebang: Python3 shebang line not working as expected

The fix, as best I can tell, would be to set -e ansible_python_interpreter=/the/path/to/the/real/python or I've seen some folks try -e 'ansible_python_interpreter="/bin/env python"' (I modified the path to env since apparently SunOS doesn't have env in /usr/bin/env but I don't have a SunOS instance to test that theory, so caveat emptor)