bashmonitoringicinga2

Icinga2 check_by_ssh plugin returns 255 without running the command

I'm configuring a Icinga2 server and want it to run local scripts on external machines using the check_by_ssh plugin, and I encountered a strange issue. I've searched for an answer for few hours, but no luck.

My command object looks as follows:

object CheckCommand "check_procs" {
        import "by_ssh"
        vars.by_ssh_logname = "root"
        vars.by_ssh_port = "22"
        vars.by_ssh_command = "/tmp/test.sh"
        vars.by_ssh_identity = "/etc/icinga2/conf.d/services/id_rsa.pub"
        vars.by_ssh_ipv4 = "true"
        vars.by_ssh_quiet = "true"
}

The content of test.sh is simply exit 0. I have a trust between my Icinga box and the remote machine I'm running the command at.

When I'm executing the command thru shell, it works

[root@icinga ~]# ssh root@10.10.10.1 -C "/tmp/test.sh"
[root@icinga ~]# echo $?
0

But when it is executed by the server, I see on my Icingaweb2 this output:

UNKNOWN - check_by_ssh: Remote command '/tmp/test.sh' returned status 255

Now I have added a touch success to test.sh script, in order to see if it is executed at all - but it seems it doesn't. That means when Icinga executes my script, it fails before even executing it.

Any clues what can it be? There are no many examples online either of check_by_ssh with Icinga2.

NOTE: Icinga uses root user to identify with the remote server. I know this is not best practice, but this is development env.

UPDATE: I think I have found the issue. The problem is that I'm trying to use root user to login the remote machine. This IS NOT supported, even with public key authentication. The script has to be executed with the user icinga

2nd Update: I got it works. The issue was keys authentication, the fact that icinga uses the user icinga to execute the command (even when using by_ssh_logname attribute) and the addition of vars.by_ssh_options = "StrictHostKeyChecking no"

Solution

  • I've found the issues, there were few of them in my case.

    1. Icinga used icinga user to login through SSH, even when I used -l root. So, to install ssh keys I had to execute ssh-copy-id icinga@HOST under root user (Icinga shell is set to /sbin/nologin)
    2. I then copied the private key (again, of the root user) to icinga folder so it is accessible for the application, and changed the ownership of the file
    3. Next, I tried to login using icinga user to the remote machine sudo -u icinga ssh icinga@HOST -i id_rsa
    4. If step 3 fails, you need to figure it before you continue. Next thing I did was adding StrictHostKeyChecking no to the module options.

    Voila, this works now.