Debian, Monitoring, Sensu

Sensu – Cloud Monitoring Tool

Monitoring always plays an important role, especially for sysadmins. There are a lot of Monitoring tools available, like Nagios, Zenoss, Icinga etc. Sensu is a new generation Cloud monitoring tool designed by Sonian. Sensu is bascially written in Ruby, uses RabbitMQ Server as the Message Broker for Message transactions, and Redis for storing the data’s.

Sensu has 3 operation Mode.

1) Request-Reply Mode, where the server will send a check request to the clients through the RabbitMQ and the clients will reply back the results.

2) Standalone Mode, where the server will not send any check request, instead the client itself will run the checks according to interval mentioned, and sends the results to the sensu master through the Result queue in RabbitMQ.

3) Push Mode, where the client will send out results to a specific handler.

So now we can start installing the dependencies for sensu, ie, RabbitMQ and Redis.

Setting up RabbitMQ

Let’s add the RabbitMQ official APT repo.

$ echo "deb http://www.rabbitmq.com/debian/ testing main" >/etc/apt/sources.list.d/rabbitmq.list

$ curl -L -o ~/rabbitmq-signing-key-public.asc http://www.rabbitmq.com/rabbitmq-signing-key-public.asc

$ apt-key add ~/rabbitmq-signing-key-public.asc && apt-get update

Now we can install RabbitMQ

$ apt-get install rabbitmq-server erlang-nox

Now we need to generate SSL certificates for RabbitMQ and the sensu clients. We can use RabbitMQ with out ssl also, but it will more secure with SSL, @joemiller has wrote a script to generate the SSL certificates. It’s avaliable in his GitHub repo. Clone the repo and modify the “openssl.cnf” according to our need and then we can go ahead with generating the certificates.

$ git clone git://github.com/joemiller/joemiller.me-intro-to-sensu.git

$ cd joemiller.me-intro-to-sensu/

$ ./ssl_certs.sh clean && /ssl_certs.sh generate

Now copy the server key and cert files to the RabbitMQ folder in “/etc/rabbitmq/”

$ mkdir /etc/rabbitmq/ssl

$ cp server_key.pem /etc/rabbitmq/ssl/

$ cp server_cert.pem /etc/rabbitmq/ssl/

$ cp testca/cacert.pem /etc/rabbitmq/ssl/

Now create the RabbitMQ config file, “/etc/rabbitmq/rabbitmq.config”, and add the following lines in it.

[
  {rabbit, [
      {ssl_listeners, [5671]},
      {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
               {certfile,"/etc/rabbitmq/ssl/server_cert.pem"},
               {keyfile,"/etc/rabbitmq/ssl/server_key.pem"},
               {verify,verify_peer},
               {fail_if_no_peer_cert,true}]}
    ]}
].

Once the config file is created, restart the RabbitmQ server. Now RabbitMQ has a cool management console, we can enable this by running ”rabbitmq-plugins enable rabbitmq_management” in console. Once the Management console is enabled, we can access it RabbitMQ Web UI: Username is “guest”, password is “guest” – http://SENSU-SERVER:55672. Protocol amqp should be bound to port 5672 and amqp/ssl on port 5671.

Now let’s create a vhost and user for Sensu in RabbitMQ.

 $ rabbitmqctl add_vhost /sensu

 $ rabbitmqctl add_user sensu mypass

 $ rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

Setting up Redis Server

Now we can set up Redis server. This will used by Sensu for stroring data’s. Ubuntu’s Apt repo ships with latest Redis server, so we can directly install it.

$ apt-get install redis-server

Installing Sensu Server

Sensu has a public repository which can be used to install the necessary sensu packages. First we need to add the repository public key.

$ wget -q http://repos.sensuapp.org/apt/pubkey.gpg -O- | sudo apt-key add -

Now add the repo sources in APT

$ echo " deb     http://repos.sensuapp.org/apt sensu main" >> /etc/apt/sources.list && apt-get update

$ apt-get install sensu

Enable the sensu services to start automatically during system startup.

$ update-rc.d sensu-server defaults

$ update-rc.d sensu-api defaults

$ update-rc.d sensu-client defaults

$ update-rc.d sensu-dashboard defaults

Copy the client ssl cert and key to /etc/sensu folder, say to a subfolder ssl.

$ cp client_key.pem client_cert.pem  /etc/sensu/ssl/

Now we need setup the sensu master, create a file ”/etc/sensu/config.json” and add the below lines.

         {
        "rabbitmq": {
          "ssl": {
            "private_key_file": "/etc/sensu/ssl/client_key.pem",
            "cert_chain_file": "/etc/sensu/ssl/client_cert.pem"
          },
          "port": 5671,
          "host": "localhost",
          "user": "sensu",
          "password": "mypass",
          "vhost": "/sensu"
        },
        "redis": {
          "host": "localhost",
          "port": 6379
        },
        "api": {
          "host": "localhost",
          "port": 4567
        },
        "dashboard": {
          "host": "localhost",
          "port": 8080,
          "user": "admin",
          "password": "sensu@123"
        },
        "handlers": {
          "default": {
            "type": "pipe",
            "command": "true"
          }
        }
      }

By default sensu package comes with all sensu-server,sensu-client,sensu-api and sensu-dashboard., If we dont want to use the current machine as a client, we can stop the sensu-client from running, and do not create the client config. But for testing purpose, i’m going to add the current machine as client also. Create a file ”/etc/sensu/conf.d/client.json” and add the client configuration in JSON format.

        {
          "client": {
          "name": "sensu.test.com",
          "address": "192.168.1.108",
          "subscriptions": [ "vmmaster" ]
         }
       }

Now restart the sensu-client to affect the changes. The logs are recorded at ”/var/log/sensu/sensu-client.log” file. We can access the sensu-dashboard from “http://SENSU SERVER:8080”, with the username and password mentioned in the config.json file.

Setting up a Separate Sensu-Client Node

If we want to setup sensu-client on a separate node, just dd the Sensu apt repo, and install the sensu package. After that just enable only the sensu-client service and remove all other sesnu-services. Then create a config.json file and add only the rabbitmq server details in it. Now generate a separate SSL certificate for the new client and use that in the config file.

       {
      "rabbitmq": {
        "ssl": {
          "private_key_file": "/etc/sensu/ssl/client1_key.pem",
          "cert_chain_file": "/etc/sensu/ssl/client1_cert.pem"
        },
        "port": 5671,
        "host": "192.168.1.108",
        "user": "sensu",
        "password": "mypass",
        "vhost": "/sensu"
      }
    }

Now create the “client.json” in the “/etc/sensu/conf.d/” folder.

        {
              "client": {
              "name": "client1.test.com",
              "address": "192.168.1.212",
              "subscriptions": [ "vmmaster" ]
             }
           }

Restart the the sensu-clinet, and check the “/var/log/sensu/sensu-client.log”, if things goes fine, we can see client connecting to the RabbitMQ server also we can see the config is getting applied.

{"timestamp":"2013-04-23T22:53:27.870728+0530","level":"warn","message":"config file applied changes","config_file":"/etc/sensu/conf.d/client.json","changes":{"client":[null,{"name":"client1.test.        com","address":"192.168.1.212","subscriptions":["vmmaster"]}]}}
{"timestamp":"2013-04-23T22:53:27.879671+0530","level":"info","message":"loaded extension","type":"mutator","name":"only_check_output","description":"returns check output"}
{"timestamp":"2013-04-23T22:53:27.883504+0530","level":"info","message":"loaded extension","type":"handler","name":"debug","description":"outputs json event data"}

Once the Sensu Server and Client are configured successfully, then we can go ahead adding the check’s. One of the best thing of sensu, all the config’s are written in JSON format, which very easy for us to create as well as to understand things. In the next blog, i will explain on how to create the check’s and how to add these check’s to various clients, and how to add handler’s like Email alerts, Sending Metrics to graphite.

Advertisements
Standard

7 thoughts on “Sensu – Cloud Monitoring Tool

  1. Pingback: beingasysadmin

  2. Adithya says:

    Hi,
    I am new to Sensu monitoring tool. I have configured check as per Sensu official website after all basic configuration, but getting an error in in the sensu-client log as,
    {“timestamp”:”2014-07-12T10:00:24.653244+0000″,”level”:”info”,”message”:”publishing check result”,”payload”:{“client”:”Client1″,”check”:{“name”:”cron_check”,”issued”:1405159224,”command”:”/etc/sensu/plugins/check-procs.rb -p crond -C 1 “,”executed”:1405159224,”output”:”/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require’: cannot load such file — sensu-plugin/check/cli (LoadError)\n\tfrom /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require’\n\tfrom /etc/sensu/plugins/check-procs.rb:29:in `’\n”,”status”:1,”duration”:0.026}}}
    Please could you help in this. Thank you.

    • Hi, could you try using the embedded ruby that comes with sensu. Go to “/etc/default/sensu” and make embedded ruby to true. The default system ruby that using, sensu plugin gem is missing. Sensu comes with latest Ruby embedded. If you really want to use system ruby, “gem install sensu-plugin” will do the trick 🙂

      • ivan says:

        Hey Deepakmdas. Thanks for a post. This is the error I get when I try to run one of the plugins:

        (1:40)$ sudo /etc/sensu/plugins/check-es-cluster-status.rb
        /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require’: no such file to load — sensu-plugin/check/cli (LoadError)
        from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require’
        from /etc/sensu/plugins/check-es-cluster-status.rb:31

        And this is my /etc/default/sensu
        (1:41)$ cat /etc/default/sensu
        EMBEDDED_RUBY=true
        LOG_DIR=STDOUT
        LOG_LEVEL=info
        SERVICE_MAX_WAIT=10

        Any suggestion is more than appreciated.

        Best

  3. Pingback: sensu – The open source monitoring framework. | digitalwhores

  4. ivan says:

    Hey Deepakmdas. Thanks for a post. This is the error I get when I try to run one of the plugins:

    (1:40)$ sudo /etc/sensu/plugins/check-es-cluster-status.rb
    /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require’: no such file to load — sensu-plugin/check/cli (LoadError)
    from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require’
    from /etc/sensu/plugins/check-es-cluster-status.rb:31

    And this is my /etc/default/sensu
    (1:41)$ cat /etc/default/sensu
    EMBEDDED_RUBY=true
    LOG_DIR=STDOUT
    LOG_LEVEL=info
    SERVICE_MAX_WAIT=10

    Any suggestion is more than appreciated.

    Best

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s