opentsdbsensu

sensu: "previous check command execution in progress"

My client-side sensu metric is reporting a WARN and the data is not getting to my OpenTSDB. It seems to be stuck, but I don't understand what the message is telling me. Can someone translate?

The command is a ruby script.

In /var/log/sensu/sensu-client.log :

{"timestamp":"2014-09-11T16:06:51.928219-0400",
   "level":"warn",
   "message":"previous check command execution in progress",
   "check":{"handler":"metric_store","type":"metric",
            "standalone":true,"command":"...",
            "output_type":"json","auto_tag_host":"yes",
            "interval":60,"description":"description here",
            "subscribers"["system"],
            "name":"foo_metric","issued":1410466011,"executed":1410465882
   }
 }

My questions:

  1. what does this message mean?
  2. what causes this?
  3. Does it really mean we are waiting for the same check to run? if so, how do we clear it?
Solution

    1. This error means that sensu is (or thinks it is, actually executing this check currently https://github.com/sensu/sensu/blob/4c36d2684f2e89a9ce811ca53de10cc2eb98f82b/lib/sensu/client.rb#L115

    2. This can be caused by stacking checks, that take longer than their interval to run. (60 seconds in this case)

    3. You can try to set the "timeout" option in the check definition: https://github.com/sensu/sensu/blob/4c36d2684f2e89a9ce811ca53de10cc2eb98f82b/lib/sensu/client.rb#L101

    To try to make sensu time out after a while on that check. You could also add internal logic to your check to make it not hang.