Debian, logstash, Monitoring

Lumberjack – a Light Weight Log Shipper for Logstash

Logstash is one of the coolest projects that i always wanted to play around. Since i’m a sysadmin, i’m forced to handle multiple apps, which will logs in different formats. The most weird part is the timestamps, where most of the app uses it’s own time formats. Logstash helps us to solve such situations, we can remodify the time stamp to a standard time format, we can use the predefined filter’s for filtering out the log’s, even we can create our own filter’s using regex. All the documentations are available in the Logstash website Logstash mainly has 3 parts, 1) INPUT -> from which the log’s are shipped to Logstash, 2) Filter -> for filtering our incoming log’s to suit to our needs, 3) Output -> For storing or relaying the Filtered output log’s to various Applications.

Lumberjack is one such input plugin designed for logstash. Though the plugin is still in beta state, i decided to give it a try. By default we can also use logstash itself for shipping logs to centralized Logstash server, the JVM made it difficult to work with many of my constrained machines. Lumberjack claims to be a light weight log shipper which uses SSL and we can add custom fields for each line of log which we ships.

Setting up Logstash Server

Download the latest the logstash jar file from the logstash website. Now create a logstash configuration file for the logstash instance. In the config file, we have to enable the lumberjack plugin. Lumberjack uses SSL CA to verify the server. So we need to generate the same for the logstash server. We can use the below mentioned command to generate the SSL certificate and key.

$ openssl req -x509 -newkey rsa:2048 -keyout /etc/ssl/logstash.key -out /etc/ssl/logstash.pub -nodes -days 3650

Below is the sample logstash conf file which i used for stashing logs from Socklog.

input {

  lumberjack {
    type => "qmail"
    port => 4545
    ssl_certificate => "/etc/ssl/logstash.pub"
        ssl_key => "/etc/ssl/logstash.key"
  }
}

filter {
  grok {
        type => "socklog"
        pattern => "%{DATA:logfacility}: %{SYSLOGTIMESTAMP:timestamp} %{DATA:program}: *"
  }
  mutate {
        replace => [ "@message", "%{mess}" ]
  }
  date {
        type => "socklog"
        match => [ "timestamp", "MMM dd HH:mm:ss" ]
  }
}

output {
  stdout {
    debug => true
      }
}

Now we can start the the logstash using the above config.

$ java -jar logstash-1.1.13-flatjar.jar agent -f logstash.conf -v

Once the logstash has started successfully, we can use netstat to check if it listening on port 4545. I’m currently running logstash in the foreground, below is the logoutput from logstash

Starting lumberjack input listener {:address=>"0.0.0.0:4545", :level=>:info}
Input registered {:plugin=><LogStash::Inputs::Lumberjack type=>"socklog", ssl_certificate=>"/etc/ssl/logstash.pub", ssl_key=>"/etc/ssl/logstash.key", charset=>"UTF-8", host=>"0.0.0.0">, :level=>:info}
Match data {:match=>{"@message"=>["%{DATA:logfacility}: %{SYSLOGTIMESTAMP:timestamp} %{DATA:program}: *"]}, :level=>:info}
Grok compile {:field=>"@message", :patterns=>["%{DATA:logfacility}: %{SYSLOGTIMESTAMP:timestamp} %{DATA:program}: *"], :level=>:info}
Output registered {:plugin=><LogStash::Outputs::Stdout debug_format=>"ruby", message=>"%{@timestamp} %{@source}: %{@message}">, :level=>:info}
All plugins are started and registered. {:level=>:info}

Setting up Lumberjack agent

On the machine from which we are going to ship the log’s, clone the Lumberjack github repo.

$ git clone https://github.com/jordansissel/lumberjack.git

Install the fpm ruby gem, which is required to build the lumberjack package.

$ gem install fpm

$ cd lumberjack && make

$ make deb   => This will build a debian package of the lumberjack

$ dpkg -i lumberjack_0.0.30_amd64.deb  => The package will install all the files to the `/opt/lumberjack`

Now copy the SSL certificate which we have generated at the Logstash server, to the Lumberjack machine. Once the SSL certificte has been copied, we can start the lumberjack agent.

$ /opt/lumberjack/bin/lumberjack --ssl-ca-path ./ssl/logstash.pub --host logstash.test.com --port 4545 /var/log/socklog/main/current

Below is the log output from the lumberjack.

2013-06-25T15:04:32.798+0530 Watching 1 files, setting open file limit to 103
2013-06-25T15:04:32.798+0530 Watching 1 files, setting memory usage limit to 1048576 bytes
2013-06-25T15:04:32.878+0530 Connecting to logstash.test.com(192.168.19.19):4545
2013-06-25T15:04:33.186+0530 slow operation (0.307 seconds): connect to 192.168.19.19:4545
2013-06-25T15:04:33.186+0530 Connected successfully to logstash.test.com(192.168.19.19):4545
2013-06-25T15:04:34.653+0530 Declaring window size of 4096
2013-06-25T15:04:36.734+0530 flushing since nothing came in over zmq

Now we will start getting the output from the Logstash in our screen, since we are using the ‘stdout’ output plugin. A very good detailed documentation about Lumberjack and Logstash can be found here, written by Brian Altenhofel. He had given a talk on this at Drupalcon 2013, Portland. The video for the talk is available here. It’s a very good blog post.

Advertisements
Standard
Debian, Monitoring, Sensu

Sensu Admin – a GUI for Sensu API

In my previous post’s, i’ve explained on How to setup Sensu server and setting up check’s and handler’s. The default dashboard is very simple with limited options, but for those who wants a full fledged dashboard, there is a Rails project in Github Sensu-Admin. So let’s try setting it up.

First clone the repository from Github.

$ git clone https://github.com/sensu/sensu-admin.git

Now go to sensu-admin folder, and run bundle install to install all the dependency gems. Now go inside the ”config” folder, edit the ”database.yml” and fill in the database details. I’m going to use mysql, below is my database config.

development:
   adapter: mysql2
   database: sensudb
   username: sensu
   password: secreto
   host: localhost
production:
   adapter: mysql2
   database: sensudb
   username: sensu
   password: secreto
   host: localhost

Now run rake db:migrate and then rake db:seed. The seed file creates auser account named ”admin@example.com” with password ”secret”.

We can start the Rails app by running “rails s”, this will start the app using the thin webserver at port 3000. Access the dashboard using the url ”http://server_ip:3000” Login to the dashboard with the admin@example.com and go to the “*Account” tab and modify the default user name and password. Now we go through tabs and check if it displays the checks, clients, events etc properly. This is a screenshot of the SensuAdmin dashboard.

Standard
Debian, Monitoring, Sensu

Sensu – Adding Check’s and Handler’s

In my previous post, i’ve explained on how to setup Sensu Server and Client. Now i’m going to explain how to setup Check’s and Handler’s in Sensu. There is a very good collection of sensu-community-plugins.

Setting up Check’s

On the Sensu Client Node,

First clone the plugins repository on the client node. Now install the ”sensu-plugin” gem on the client node. And then copy the required plugins to /etc/sensu/plugins/ folder.

On the Sensu Server,

We need to define the check first. Create a json config file for the check in /etc/sensu/conf.d. Following is a sample check config,

     {
    "checks": {
         "snmp_check": {
         "handlers": ["default"],
         "command": "/etc/sensu/plugins/check-snmp.rb -w 10 -c 20",
         "interval": 30,
         "subscribers": [ "snmp" ]
          }
      }
   }

The above check will be applied to all clients subscribed to ”snmp” exchange. Based on the interval, Server will publish this check request, which will reach all the clients subscribed to the ”snmp” exchange using an arbitrary queue. The client will run the command mentioned in the command part, and then it will publish the result back to th server through Result queue. The check_snmp is a small plugin written by me. If we check the sensu-server log, we can see the result coming from the client machine. Below one is a similar log output in my sensu-server log.

{"timestamp":1366968018},"check":{"handlers":["default","mailer"],"command":"/etc/sensu/plugins/check-snmp.rb -w 1 -c 3","interval":100,"subscribers":["snmp"],"name":"snmp_check","issued":1366968407,"executed":1366968028,"output":"CheckSNMP WARNING: Warning state detected\n","status":1,"duration":0.526,"history":["0","0","1"]},"occurrences":1,"action":"create"},"handler":{"type":"pipe","command":"true","name":"default"}}

The above log line shows us what are handler’s enabled for this check, what is the executed command, subcribers, name of the check, timestamp at the time when the command was issued, timestamp of the time when the server has received the result, Output of the check command etc. If there is any while executing th check command, we can see the errors popping in the log’s soon after this line in the server log.

Setting up Handler’s

Sensu has got a very good collection Handler’s, available at the sensu-community-plugin repo in github. For example there is a hanlder called ”show”, available at the debug section in Handler’s, which will display a more debug report about the Event as well as the Sensu server’s settings. This is the output which i got after applying ”show” handler in my serverlog. But it’s not possible to go check the log’s continously, so there another plugin called “mailer”, which can send email alerts like how nagios does.

So first get the “mailer” plugin files from the sensu-community-plugin repo in github.

wget -O /etc/sensu/handlers/mailer.rb https://raw.github.com/sensu/sensu-community-plugins/master/handlers/notification/mailer.rb
wget -O /etc/sensu/conf.d/mailer.json https://raw.github.com/sensu/sensu-community-plugins/master/handlers/notification/mailer.json

Now edit the mailer.json, and change the settings to fit to our environment. We need to define a new pipe handler for this new handler. So create a file /etc/sensu/conf.d/handler_mailer.json, and add the below lines to it.

        {
    "handlers": {
        "mailer": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/mailer.rb"
        }
          }
      }

Now go to the one of the check config files, where we want to apply this new “mailer” handler.

           {
    "checks": {
         "snmp_check": {
         "handlers": ["default", "mailer"],         
         "command": "/etc/sensu/plugins/check-snmp.rb -w 10 -c 20",
         "interval": 30,
         "subscribers": [ "snmp" ]
          }
      }
   }

Now restart the sensu-server to make the new changes to come into effect. If everything goes fine, when the sensu detects a state change it will execute this mailer handler, we can also see the below lines in server log.

"action":"create"},"handler":{"type":"pipe","command":"/etc/sensu/handlers/mailer.rb","name":"mailer"

Sensu is executing the mailer script, and if there is any problem, we will see the corresponding error following the above line, or we will receive the email alert to email id mentioned in the “mailer.json” file. But in my case, i was getting an error, when the sensu invoked the “mailer” handler.

{"timestamp":"2013-04-25T15:03:32.002132+0530","level":"info","message":"/etc/sensu/handlers/mailer.rb:28:in `handle': undefined method `[]' for nil:NilClass (NoMethodError)"}
{"timestamp":"2013-04-25T15:03:32.002308+0530","level":"info","message":"\tfrom /var/lib/gems/1.9.1/gems/sensu-plugin-0.1.7/lib/sensu-handler.rb:41:in `block in <class:Handler>'"}

After playing for some time, i came to know that, it was not parsing the options from the mailer.json file, so i manually added the smtp and email settings directly in mailer.rb file. Then it started working fine. I’m writing a small script which will be using the basic ‘net/smtp’ library to send out mails. There are many other cool Handler’s like sending matrices to Graphite, Logstash, Graylog, sending notifcations to irc,xmpp,campfire etc. Compare to traditional monitoring tools, Sensu is an Amazing tool, we can use any check script’s, whether it’s ruby or perl or bash, doesn’t matter. The one common thing which i heard about other people, was the lack of proper dashboard like the traditional monitoring tools. Though Sensu dashboard is a simple one, i’m sure it will improve a lot in future.

Since I’m a CLI Junky, I dont care much about the dashboard thing, apart from that i have many good and interesting stuffs to hang around with Sensu. Cheers to portertech and sonian for open sourcing such an amazing tool.

Standard
Debian, Monitoring, Sensu

Sensu – Cloud Monitoring Tool

Monitoring always plays an important role, especially for sysadmins. There are a lot of Monitoring tools available, like Nagios, Zenoss, Icinga etc. Sensu is a new generation Cloud monitoring tool designed by Sonian. Sensu is bascially written in Ruby, uses RabbitMQ Server as the Message Broker for Message transactions, and Redis for storing the data’s.

Sensu has 3 operation Mode.

1) Request-Reply Mode, where the server will send a check request to the clients through the RabbitMQ and the clients will reply back the results.

2) Standalone Mode, where the server will not send any check request, instead the client itself will run the checks according to interval mentioned, and sends the results to the sensu master through the Result queue in RabbitMQ.

3) Push Mode, where the client will send out results to a specific handler.

So now we can start installing the dependencies for sensu, ie, RabbitMQ and Redis.

Setting up RabbitMQ

Let’s add the RabbitMQ official APT repo.

$ echo "deb http://www.rabbitmq.com/debian/ testing main" >/etc/apt/sources.list.d/rabbitmq.list

$ curl -L -o ~/rabbitmq-signing-key-public.asc http://www.rabbitmq.com/rabbitmq-signing-key-public.asc

$ apt-key add ~/rabbitmq-signing-key-public.asc && apt-get update

Now we can install RabbitMQ

$ apt-get install rabbitmq-server erlang-nox

Now we need to generate SSL certificates for RabbitMQ and the sensu clients. We can use RabbitMQ with out ssl also, but it will more secure with SSL, @joemiller has wrote a script to generate the SSL certificates. It’s avaliable in his GitHub repo. Clone the repo and modify the “openssl.cnf” according to our need and then we can go ahead with generating the certificates.

$ git clone git://github.com/joemiller/joemiller.me-intro-to-sensu.git

$ cd joemiller.me-intro-to-sensu/

$ ./ssl_certs.sh clean && /ssl_certs.sh generate

Now copy the server key and cert files to the RabbitMQ folder in “/etc/rabbitmq/”

$ mkdir /etc/rabbitmq/ssl

$ cp server_key.pem /etc/rabbitmq/ssl/

$ cp server_cert.pem /etc/rabbitmq/ssl/

$ cp testca/cacert.pem /etc/rabbitmq/ssl/

Now create the RabbitMQ config file, “/etc/rabbitmq/rabbitmq.config”, and add the following lines in it.

[
  {rabbit, [
      {ssl_listeners, [5671]},
      {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
               {certfile,"/etc/rabbitmq/ssl/server_cert.pem"},
               {keyfile,"/etc/rabbitmq/ssl/server_key.pem"},
               {verify,verify_peer},
               {fail_if_no_peer_cert,true}]}
    ]}
].

Once the config file is created, restart the RabbitmQ server. Now RabbitMQ has a cool management console, we can enable this by running ”rabbitmq-plugins enable rabbitmq_management” in console. Once the Management console is enabled, we can access it RabbitMQ Web UI: Username is “guest”, password is “guest” – http://SENSU-SERVER:55672. Protocol amqp should be bound to port 5672 and amqp/ssl on port 5671.

Now let’s create a vhost and user for Sensu in RabbitMQ.

 $ rabbitmqctl add_vhost /sensu

 $ rabbitmqctl add_user sensu mypass

 $ rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

Setting up Redis Server

Now we can set up Redis server. This will used by Sensu for stroring data’s. Ubuntu’s Apt repo ships with latest Redis server, so we can directly install it.

$ apt-get install redis-server

Installing Sensu Server

Sensu has a public repository which can be used to install the necessary sensu packages. First we need to add the repository public key.

$ wget -q http://repos.sensuapp.org/apt/pubkey.gpg -O- | sudo apt-key add -

Now add the repo sources in APT

$ echo " deb     http://repos.sensuapp.org/apt sensu main" >> /etc/apt/sources.list && apt-get update

$ apt-get install sensu

Enable the sensu services to start automatically during system startup.

$ update-rc.d sensu-server defaults

$ update-rc.d sensu-api defaults

$ update-rc.d sensu-client defaults

$ update-rc.d sensu-dashboard defaults

Copy the client ssl cert and key to /etc/sensu folder, say to a subfolder ssl.

$ cp client_key.pem client_cert.pem  /etc/sensu/ssl/

Now we need setup the sensu master, create a file ”/etc/sensu/config.json” and add the below lines.

         {
        "rabbitmq": {
          "ssl": {
            "private_key_file": "/etc/sensu/ssl/client_key.pem",
            "cert_chain_file": "/etc/sensu/ssl/client_cert.pem"
          },
          "port": 5671,
          "host": "localhost",
          "user": "sensu",
          "password": "mypass",
          "vhost": "/sensu"
        },
        "redis": {
          "host": "localhost",
          "port": 6379
        },
        "api": {
          "host": "localhost",
          "port": 4567
        },
        "dashboard": {
          "host": "localhost",
          "port": 8080,
          "user": "admin",
          "password": "sensu@123"
        },
        "handlers": {
          "default": {
            "type": "pipe",
            "command": "true"
          }
        }
      }

By default sensu package comes with all sensu-server,sensu-client,sensu-api and sensu-dashboard., If we dont want to use the current machine as a client, we can stop the sensu-client from running, and do not create the client config. But for testing purpose, i’m going to add the current machine as client also. Create a file ”/etc/sensu/conf.d/client.json” and add the client configuration in JSON format.

        {
          "client": {
          "name": "sensu.test.com",
          "address": "192.168.1.108",
          "subscriptions": [ "vmmaster" ]
         }
       }

Now restart the sensu-client to affect the changes. The logs are recorded at ”/var/log/sensu/sensu-client.log” file. We can access the sensu-dashboard from “http://SENSU SERVER:8080”, with the username and password mentioned in the config.json file.

Setting up a Separate Sensu-Client Node

If we want to setup sensu-client on a separate node, just dd the Sensu apt repo, and install the sensu package. After that just enable only the sensu-client service and remove all other sesnu-services. Then create a config.json file and add only the rabbitmq server details in it. Now generate a separate SSL certificate for the new client and use that in the config file.

       {
      "rabbitmq": {
        "ssl": {
          "private_key_file": "/etc/sensu/ssl/client1_key.pem",
          "cert_chain_file": "/etc/sensu/ssl/client1_cert.pem"
        },
        "port": 5671,
        "host": "192.168.1.108",
        "user": "sensu",
        "password": "mypass",
        "vhost": "/sensu"
      }
    }

Now create the “client.json” in the “/etc/sensu/conf.d/” folder.

        {
              "client": {
              "name": "client1.test.com",
              "address": "192.168.1.212",
              "subscriptions": [ "vmmaster" ]
             }
           }

Restart the the sensu-clinet, and check the “/var/log/sensu/sensu-client.log”, if things goes fine, we can see client connecting to the RabbitMQ server also we can see the config is getting applied.

{"timestamp":"2013-04-23T22:53:27.870728+0530","level":"warn","message":"config file applied changes","config_file":"/etc/sensu/conf.d/client.json","changes":{"client":[null,{"name":"client1.test.        com","address":"192.168.1.212","subscriptions":["vmmaster"]}]}}
{"timestamp":"2013-04-23T22:53:27.879671+0530","level":"info","message":"loaded extension","type":"mutator","name":"only_check_output","description":"returns check output"}
{"timestamp":"2013-04-23T22:53:27.883504+0530","level":"info","message":"loaded extension","type":"handler","name":"debug","description":"outputs json event data"}

Once the Sensu Server and Client are configured successfully, then we can go ahead adding the check’s. One of the best thing of sensu, all the config’s are written in JSON format, which very easy for us to create as well as to understand things. In the next blog, i will explain on how to create the check’s and how to add these check’s to various clients, and how to add handler’s like Email alerts, Sending Metrics to graphite.

Standard
Monitoring

Monitoring with ZENOSS

It’s being a year since i have really played with Centos or any Redhat based Distro’s. I saw a few videos on youtube realting to zenoss, which is a new generation monitoring tool. Later i attended two zenoss webinar’s, which made to try it out in own infrastructure. In this blog i will explain how to setup zenoss on a Centos6.4 machine. Make sure that you have atleast 2GB of Ram. Initially i put 1GB of Ram and 2GB of swap in my Centos VM. But when i started the zenoss services, the whole and ram and swap was consumed and finaly i was not able to start the services.

Basicaly zenoss need RabbitMQ messaging server, JAVA6, MYSQL as its dependencies. There is an automated script available from the zenos website, which will download and install all necessary dependencies. It’s a bash script. We can download it from the below link.

$  wget --no-check-certificate https://github.com/zenoss/core-autodeploy/tarball/4.2.3 -O auto.tar.gz

Once we extract the above tar ball, we can see a bunch of files. zenpack_actions.txt file contains the list of zenpacks which is going to be installed. We can modify it based on our needs.

Once done, we can start the installer script.

$ ./core-autodeploy.sh

This will start by downloading the zenoss rpm file. Once the installation completed, it was giving an error, saying that “connection reset” while installing the zenpacks. I was going through all the log files, finally i found that the error was in the rabbitmq. The zenoss user authentication was failing. Below is the error which iwas getting in the rabbitmq log.

=INFO REPORT==== 10-Apr-2013::09:37:00 ===
accepting AMQP connection <0.3533.0> (127.0.0.1:38662 -> 127.0.0.1:5672)

=ERROR REPORT==== 10-Apr-2013::09:37:03 ===
closing AMQP connection <0.3533.0> (127.0.0.1:38662 -> 127.0.0.1:5672):
{channel0_error,starting,
            {amqp_error,access_refused,
                        "PLAIN login refused: user 'zenoss' - invalid credentials",
                        'connection.start_ok'}}

The error says that the zenoss user credential is wrong. So using “rabbitmqctl” command i reset the zenoss user password. Once the password is changed, we have to mention the new passowrd in the zenoss global.conf file. This file will be present in “/opt/zenoss/etc” location. Open the the global.conf file, and replace the amqppassword with the new password. By default during installation, the script generates a base64 encoded random password using the openssl. Once we hab=ve replaced the password, we can start the zenoss service.

$ service zenoss start

Now while starting the service, zenoss will continue the installing the zenpacks. Once the service is started, we can access the WebGUI from http://server_ip:8080 url. Initially it will ask us to set the password for the admin user as well as to create a secondary user. More over it will ask us to add hosts to monitor, we can skip this step and move the dashboard. Later on we can add the hosts directly from the infrastructure tab. I’ve added my vps as well as few of my local server’s with snmp, so far it is working perfectly. There many cool stuffs inside zenoss, hope this will a cool playground …

Standard
Debian, Monitoring

Using SNMP With Icinga

In my previous [blog](https://beingasysadmin.wordpress.com/2013/02/27/monitoring-servers-with-icinga-2/) i explained how to set up the Icinga monitoring system on Debian based machine. In this i will be explaining on how to use SNMP with icinga. Using SNMP we can get various info from a remote machine which can be used to icinga.

On the remote server we need to install the snmp server. In ordr to check whether the snmp is working fine, we can install the snmp client also on the same machine.

apt-get install snmp snmp-server

From Debian Wheezy onwards, the snmp-mibs package has been removed from the debian repositories. But we can download the debian file from the squeeze repo. Since it has only a few depenencies and they can be installed from the debian repositories, we can manually install the mibs package. But in Ubuntu mibs package is still available in their repositories.

Once the package is installed we can run the download-mibs to install all the necessary mibs. Now once the snmpd package is installed, we need to comment the MIBS option in the /etc/snmp/snmp.conf. And define the basic settings like, location, contact etc in the snmpd.conf file. We can define the ip to which snmp should listen either in the snmpd.conf file or in the /etc/default/snmp file as an extra option. Once the settings are modified, we can start the snmpd service.

So now the service will listening on the port 631 on the ip which we defined, by default 127.0.0.1. We can also enable authentication for snmp service, so that the snmp clients which passes the authentication will be able to get the result from the snmp service.

Once the snmp service has started on the remote server, we can test the snmp onnectivity from our icinga server using the nagios’ check_snmp plugin. Once we are able to get results from the snmp service, we can deploy tese snmp services into our icinga host configurations to get the results from the remote server’s using the check_snmp command.

Standard
Debian, Monitoring

Monitoring Server’s with Icinga

Last week we had severe outage in our US vps. The load was getting too high in the server, since it’s a hosting server, and so many of our clients are hosting their DNS, website and mail server in it. So i decided to implement a monitoring system. And i decided it to extend it to all of our client server’s so that we can have a fully fledged monitoring system in our company. I’m a great fan of Sensu and i play with it regularly, but this time i decided to setup Icinga monitoring system, which can be used any System admin very easily. In this post i will be explaining on how to setup the icinga with it’s newest web interface called icinga-web. This setup works perfectly in Debian based OS. In my next post, i will be explaining on how to setup SNMP to work with icinga.

By default Ubuntu and Debian repositories has icinga packages.

$ apt-get install icinga icinga-cgi icinga-common icinga-core icinga-idoutils icinga-web icinga-web-pnp mysql-server

Once the installation is completed, by default icinga has already created a config file for the local host, which can be found in /etc/icinga/objects/. Now we can access the default icinga web interface with http://yoursystemip/icinga. The default user will be icingaadmin and password will be the default password which we have setup during the installation. Now we can create the config files for other hosts and we should restart the icinga service. This will add the host’s to our default web interface. All the check commands are defined in the /etc/nagios-plugins/config/ folder.

Now we need to set a contact so that icinga can send alerts to the specified email id. In the /etc/icinga/objects/ there is a file called contacts_icinga.cfg, where we have to define the contact info.

define contact{
    contact_name                    your_contact_name
    alias                           alias
    service_notification_period     24x7
    host_notification_period        24x7
    service_notification_options    w,u,c,r
    host_notification_options       d,r
    service_notification_commands   notify-service-by-email
    host_notification_commands      notify-host-by-email
    email                           your_contact_email_id
    }

The notify-service-by-email and notify-host-by-email commands are defined in the /etc/icinga/commands.cfg file. We can make changes to the format of the email alert by modifying this file. the “icinga-web” package will setup the basic config for the new icinga-web interface. Before starting the new interface we need to check a few settings for the new interface. First is the Database for the new interface. Ensure that the DB icinga-web and a user called icinga-web is created in the mysql. Now we need ensure that the database settings are correctly mentioned in the icinga-web settings. The settings are available in /etc/icinga-web/conf.d/. Now ensure that the database,user, and password are correctly mentioned in the databases.xml file. Now we need to ensure that the broker modules are enabled in the icinga.cfg file. comment out the below line in the icinga.cfg file to enable the idmod to enable the idomod broker module.

broker_module=/usr/lib/icinga/idomod.so config_file=/etc/icinga/idomod.cfg

Also increase the log level to debug in both icinga as well as idomod, which helps to identify error if any. Now restart the icinga and id02db services. Now go to the new interface by going to the following url, http://ip/icinga-web. the default user name is “root” and the passwod will be the one which we have given during installation

Standard