Debian, logstash, Monitoring

Lumberjack – a Light Weight Log Shipper for Logstash

Logstash is one of the coolest projects that i always wanted to play around. Since i’m a sysadmin, i’m forced to handle multiple apps, which will logs in different formats. The most weird part is the timestamps, where most of the app uses it’s own time formats. Logstash helps us to solve such situations, we can remodify the time stamp to a standard time format, we can use the predefined filter’s for filtering out the log’s, even we can create our own filter’s using regex. All the documentations are available in the Logstash website Logstash mainly has 3 parts, 1) INPUT -> from which the log’s are shipped to Logstash, 2) Filter -> for filtering our incoming log’s to suit to our needs, 3) Output -> For storing or relaying the Filtered output log’s to various Applications.

Lumberjack is one such input plugin designed for logstash. Though the plugin is still in beta state, i decided to give it a try. By default we can also use logstash itself for shipping logs to centralized Logstash server, the JVM made it difficult to work with many of my constrained machines. Lumberjack claims to be a light weight log shipper which uses SSL and we can add custom fields for each line of log which we ships.

Setting up Logstash Server

Download the latest the logstash jar file from the logstash website. Now create a logstash configuration file for the logstash instance. In the config file, we have to enable the lumberjack plugin. Lumberjack uses SSL CA to verify the server. So we need to generate the same for the logstash server. We can use the below mentioned command to generate the SSL certificate and key.

$ openssl req -x509 -newkey rsa:2048 -keyout /etc/ssl/logstash.key -out /etc/ssl/logstash.pub -nodes -days 3650

Below is the sample logstash conf file which i used for stashing logs from Socklog.

input {

  lumberjack {
    type => "qmail"
    port => 4545
    ssl_certificate => "/etc/ssl/logstash.pub"
        ssl_key => "/etc/ssl/logstash.key"
  }
}

filter {
  grok {
        type => "socklog"
        pattern => "%{DATA:logfacility}: %{SYSLOGTIMESTAMP:timestamp} %{DATA:program}: *"
  }
  mutate {
        replace => [ "@message", "%{mess}" ]
  }
  date {
        type => "socklog"
        match => [ "timestamp", "MMM dd HH:mm:ss" ]
  }
}

output {
  stdout {
    debug => true
      }
}

Now we can start the the logstash using the above config.

$ java -jar logstash-1.1.13-flatjar.jar agent -f logstash.conf -v

Once the logstash has started successfully, we can use netstat to check if it listening on port 4545. I’m currently running logstash in the foreground, below is the logoutput from logstash

Starting lumberjack input listener {:address=>"0.0.0.0:4545", :level=>:info}
Input registered {:plugin=><LogStash::Inputs::Lumberjack type=>"socklog", ssl_certificate=>"/etc/ssl/logstash.pub", ssl_key=>"/etc/ssl/logstash.key", charset=>"UTF-8", host=>"0.0.0.0">, :level=>:info}
Match data {:match=>{"@message"=>["%{DATA:logfacility}: %{SYSLOGTIMESTAMP:timestamp} %{DATA:program}: *"]}, :level=>:info}
Grok compile {:field=>"@message", :patterns=>["%{DATA:logfacility}: %{SYSLOGTIMESTAMP:timestamp} %{DATA:program}: *"], :level=>:info}
Output registered {:plugin=><LogStash::Outputs::Stdout debug_format=>"ruby", message=>"%{@timestamp} %{@source}: %{@message}">, :level=>:info}
All plugins are started and registered. {:level=>:info}

Setting up Lumberjack agent

On the machine from which we are going to ship the log’s, clone the Lumberjack github repo.

$ git clone https://github.com/jordansissel/lumberjack.git

Install the fpm ruby gem, which is required to build the lumberjack package.

$ gem install fpm

$ cd lumberjack && make

$ make deb   => This will build a debian package of the lumberjack

$ dpkg -i lumberjack_0.0.30_amd64.deb  => The package will install all the files to the `/opt/lumberjack`

Now copy the SSL certificate which we have generated at the Logstash server, to the Lumberjack machine. Once the SSL certificte has been copied, we can start the lumberjack agent.

$ /opt/lumberjack/bin/lumberjack --ssl-ca-path ./ssl/logstash.pub --host logstash.test.com --port 4545 /var/log/socklog/main/current

Below is the log output from the lumberjack.

2013-06-25T15:04:32.798+0530 Watching 1 files, setting open file limit to 103
2013-06-25T15:04:32.798+0530 Watching 1 files, setting memory usage limit to 1048576 bytes
2013-06-25T15:04:32.878+0530 Connecting to logstash.test.com(192.168.19.19):4545
2013-06-25T15:04:33.186+0530 slow operation (0.307 seconds): connect to 192.168.19.19:4545
2013-06-25T15:04:33.186+0530 Connected successfully to logstash.test.com(192.168.19.19):4545
2013-06-25T15:04:34.653+0530 Declaring window size of 4096
2013-06-25T15:04:36.734+0530 flushing since nothing came in over zmq

Now we will start getting the output from the Logstash in our screen, since we are using the ‘stdout’ output plugin. A very good detailed documentation about Lumberjack and Logstash can be found here, written by Brian Altenhofel. He had given a talk on this at Drupalcon 2013, Portland. The video for the talk is available here. It’s a very good blog post.

Advertisements
Standard

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s