Debian, Monitoring, Sensu

Sensu – Adding Check’s and Handler’s

In my previous post, i’ve explained on how to setup Sensu Server and Client. Now i’m going to explain how to setup Check’s and Handler’s in Sensu. There is a very good collection of sensu-community-plugins.

Setting up Check’s

On the Sensu Client Node,

First clone the plugins repository on the client node. Now install the ”sensu-plugin” gem on the client node. And then copy the required plugins to /etc/sensu/plugins/ folder.

On the Sensu Server,

We need to define the check first. Create a json config file for the check in /etc/sensu/conf.d. Following is a sample check config,

     {
    "checks": {
         "snmp_check": {
         "handlers": ["default"],
         "command": "/etc/sensu/plugins/check-snmp.rb -w 10 -c 20",
         "interval": 30,
         "subscribers": [ "snmp" ]
          }
      }
   }

The above check will be applied to all clients subscribed to ”snmp” exchange. Based on the interval, Server will publish this check request, which will reach all the clients subscribed to the ”snmp” exchange using an arbitrary queue. The client will run the command mentioned in the command part, and then it will publish the result back to th server through Result queue. The check_snmp is a small plugin written by me. If we check the sensu-server log, we can see the result coming from the client machine. Below one is a similar log output in my sensu-server log.

{"timestamp":1366968018},"check":{"handlers":["default","mailer"],"command":"/etc/sensu/plugins/check-snmp.rb -w 1 -c 3","interval":100,"subscribers":["snmp"],"name":"snmp_check","issued":1366968407,"executed":1366968028,"output":"CheckSNMP WARNING: Warning state detected\n","status":1,"duration":0.526,"history":["0","0","1"]},"occurrences":1,"action":"create"},"handler":{"type":"pipe","command":"true","name":"default"}}

The above log line shows us what are handler’s enabled for this check, what is the executed command, subcribers, name of the check, timestamp at the time when the command was issued, timestamp of the time when the server has received the result, Output of the check command etc. If there is any while executing th check command, we can see the errors popping in the log’s soon after this line in the server log.

Setting up Handler’s

Sensu has got a very good collection Handler’s, available at the sensu-community-plugin repo in github. For example there is a hanlder called ”show”, available at the debug section in Handler’s, which will display a more debug report about the Event as well as the Sensu server’s settings. This is the output which i got after applying ”show” handler in my serverlog. But it’s not possible to go check the log’s continously, so there another plugin called “mailer”, which can send email alerts like how nagios does.

So first get the “mailer” plugin files from the sensu-community-plugin repo in github.

wget -O /etc/sensu/handlers/mailer.rb https://raw.github.com/sensu/sensu-community-plugins/master/handlers/notification/mailer.rb
wget -O /etc/sensu/conf.d/mailer.json https://raw.github.com/sensu/sensu-community-plugins/master/handlers/notification/mailer.json

Now edit the mailer.json, and change the settings to fit to our environment. We need to define a new pipe handler for this new handler. So create a file /etc/sensu/conf.d/handler_mailer.json, and add the below lines to it.

        {
    "handlers": {
        "mailer": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/mailer.rb"
        }
          }
      }

Now go to the one of the check config files, where we want to apply this new “mailer” handler.

           {
    "checks": {
         "snmp_check": {
         "handlers": ["default", "mailer"],         
         "command": "/etc/sensu/plugins/check-snmp.rb -w 10 -c 20",
         "interval": 30,
         "subscribers": [ "snmp" ]
          }
      }
   }

Now restart the sensu-server to make the new changes to come into effect. If everything goes fine, when the sensu detects a state change it will execute this mailer handler, we can also see the below lines in server log.

"action":"create"},"handler":{"type":"pipe","command":"/etc/sensu/handlers/mailer.rb","name":"mailer"

Sensu is executing the mailer script, and if there is any problem, we will see the corresponding error following the above line, or we will receive the email alert to email id mentioned in the “mailer.json” file. But in my case, i was getting an error, when the sensu invoked the “mailer” handler.

{"timestamp":"2013-04-25T15:03:32.002132+0530","level":"info","message":"/etc/sensu/handlers/mailer.rb:28:in `handle': undefined method `[]' for nil:NilClass (NoMethodError)"}
{"timestamp":"2013-04-25T15:03:32.002308+0530","level":"info","message":"\tfrom /var/lib/gems/1.9.1/gems/sensu-plugin-0.1.7/lib/sensu-handler.rb:41:in `block in <class:Handler>'"}

After playing for some time, i came to know that, it was not parsing the options from the mailer.json file, so i manually added the smtp and email settings directly in mailer.rb file. Then it started working fine. I’m writing a small script which will be using the basic ‘net/smtp’ library to send out mails. There are many other cool Handler’s like sending matrices to Graphite, Logstash, Graylog, sending notifcations to irc,xmpp,campfire etc. Compare to traditional monitoring tools, Sensu is an Amazing tool, we can use any check script’s, whether it’s ruby or perl or bash, doesn’t matter. The one common thing which i heard about other people, was the lack of proper dashboard like the traditional monitoring tools. Though Sensu dashboard is a simple one, i’m sure it will improve a lot in future.

Since I’m a CLI Junky, I dont care much about the dashboard thing, apart from that i have many good and interesting stuffs to hang around with Sensu. Cheers to portertech and sonian for open sourcing such an amazing tool.

Advertisements
Standard

One thought on “Sensu – Adding Check’s and Handler’s

  1. Pingback: sensu – The open source monitoring framework. | digitalwhores

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s