Elasticsearch, Kibana, logstash, Monitoring, Plivo, SIP, Ubuntu, Voip

Extending ELK Stack to VOIP Infrastructure

Being a DevOps guy, i always love metrics. Visualized metrics gives a good picture of what’s happening in our live battle stations. There are now a quite lot of Open Source tools for monitoring and visualizing. It’s more than a year since i’ve started using Logstash. It never turned me down. ElasticSearch-Logstash-Kibana (ELK) is a killer combination. Though i started Elasticsearch + Logstash as a log analyzer, later StatsD and Graphite took it to the next level. When we have a simple infrastructure it’s easy to monitor. But when the infra starts scaling, it becomes quite difficult to keep track of all the events happening inside each nodes. Though service checks can help, but there is still limitation for it. I faced a lot of scenarios where things breaks but service checks will be fine. Under such scenarios logs are the only hope. They have all these events captured.

At Plivo, we manage a variety of servers from SIP, Media, Proxy, WebServers, DB’s etc. Being a fully Cloud based system, i really wanted to have a system which can keep track of all the live events/status of what’s really happening inside our infra. So my plan was to collect two important stats, 1) Server’s events 2) Application events.

Collectd and Logstash

Collectd is a daemon which collects system performance statistics periodically. Since we have a lot Server’s which handle Realtime Media, it’s a very critical component for us. We need to ensure that the server’s are not getting overloaded and there is no latency in network. I’ve been using Logstash heavily for stashing all my logs. And there is a stable input plugin for collectd to send the all the system metrics to logstash.

First we need to enable the Network Plugin, and then we need to mention our Logstash server IP and port so that collectd can start injecting metrics. Below is a sample colectd configuration.

Hostname    "test.plivo.com"
Interval 10
Timeout 4
Include "/etc/collectd/filters.conf"
Include "/etc/collectd/thresholds.conf"
ReportStats true
    LogLevel info
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin network
<Plugin interface>
    Interface "eth0"
    IgnoreSelected false
</Plugin>
<Plugin network>
    Server "{logstash_server_ip}" "logstash_server_port"    # if no port number is mentioned, it will take the default port number (25826)
</Plugin>

Now on the Logstash server, we need to add the CollectD plugin on to the input filter in the logstash’s config file.

input {
      collectd {
      port => "5555"    # default port is 25826
      }
}

Now we are set. Based the plugins enabled in the collectd config file, collctd will start sending the metrics to Logstash on the Interval mentioned in the config, default is 10s. So in my case, i wanted the Load, CPU usage, Memory usage, Bandiwdth (TX and RX) etc. There are default plugins for all these metrics, which we can just enable it in the config file. We also had some custom plugins to collect some custom metrics. BTW writing custom plugin is pretty easy in Collectd.

Now using the Logstash’s Elasticsearch output plugin, we can keep these metrics in Elasticsearch. Now this where Kibana comes in. We can start visualizing these metrics via Kibana. We need to create a custom Lucene Query. Once we have the query, we can create a custom histogram’s for each of these queries. Below aresome sample Lucene queries that we can use with Kibana.

For Load -> collectd_type:"load" AND host:"test.plivo.com"
For Network usage -> collectd_type:"if_octets" AND host:"test.plivo.com"

Below is the screenshot of histogram for Load and Network (TX and RX)

Log Events

Now next is to collect the events from the application logs. We use SIP protocol for all our VOIP sessions. So all our SIP server’s are very critical for us. SIP is pretty similar to HTTP. The response codes are very similar to HTTP responses, ie 1xx, 2xx, 3xx, 4xx, 5xx, 6xx. So i wrote some custom grok patterns so keep track of all of these responses and stores the same on the Elasticsearch.

The second stats which i was interested was our SIP registrar server. We provide SIP endpoints to our customers so that they can use the same with SIP/Soft phones. So i was more interested on stats like Number of registrations/sec, Auth error rates. Plus using ElasticSearch’s MAP facet’s i can create BetterMap. In my previous blog post’s i’ve mentioned on how to create these bettermaps using Kibana and Elasticsearch. Below bettermap screenshot shows us the SIP endpoint registrations from various locations in the last 2 hours.

Now using the Kibana we can start visualizing all these data’s. Below is a sample of Dashboard that i’ve created using Kibana.

ELK stack proved to be an amazing combination. We are currently injecting 3 million events every day and ElasticSearch was blazingly fast in indexing all theses.

Standard
HipChat, Plivo, Redis, Robut

Make Text to Speech Calls From Hip Chat

I’ve been using HipChat for the last one month. Before that it was IRC always. But when i joined Plivo’s DevOps family, i’ve got a lot of places to integrate plivo to automate a lot of stuffs. This i was planning to do some thing with Hipchat. There are a couple of HipChat bot’s available, i decided to use ’robut’, as it is simple, written in Ruby and easy to write plugins also. Robut need only one “Chatfile” as its config file. All the Hip Chat account and room details are mentioned in this file. The Plugins are enabled in this file.

First we need to clone the repository.

$ git clone https://github.com/justinweiss/robut.git

Now i need ’redis’ and ’plivo’ gems for my new plugin. So need to add the same in the Gemfile. Once the gems are added in to the Gemfile, we need to do a ”bundle install” to install all the necessary gems. The path for the Chatfile can be mentioned in ”bin/robut” file. Now before starting robut we need to add the basic HipChat settings ie, jabberid, nick, password and room. Now wen eed to create a plugin to use with robut. The default plugins are available in ”lib/robut/plugin/” folder. Below is the plugin which i created for making Text To Speech calls.

require 'json'
require 'timeout'
require 'securerandom'
require 'plivo'
require 'redis'

class Robut::Plugin::Call
  include Robut::Plugin

  def usage
    "#{at_nick} call <number> <message> "
  end

  def handle(time, sender_nick, message)
    new_msg = message.split(' ')
    if sent_to_me?(message) && new_msg[1] == 'call'
    num = new_msg[2]
    textt = message.split(' ').drop(3).join(' ')
      begin
        reply("Calling #{num}")
        plivo_auth_id = "XXXXXXXXXXXXXXXXX"
        plivo_auth_token = "XXXXXXXXXXXXXXXXX"
        uuid = SecureRandom.hex
        r = Redis.new(:host => "redis_host", :port => redis_port, :password => "redis_port")
        temp = {
               'text' => "#{textt}"
               }
        plivo_number = "plivo_number"
        to_number = "#{num}"
        answer_url = "http://polar-everglades-1062.herokuapp.com/#{uuid}"

        call_params = {
                      'from' => plivo_number,
                      'to' => to_number,
                      'answer_url' => answer_url,
                      'answer_method' => 'GET'
                      }
        r.set "#{uuid}", temp.to_json
        r.expire "#{uuid}", 3000
        sleep 5
        puts "Initiating plivo call"
        p = Plivo::RestAPI.new(plivo_auth_id, plivo_auth_token)
        details = p.make_call(call_params)
        reply("Summary #{details}")
      rescue
        reply("Sorry Unable to Make initiate the Call.")
      end
    end
  end
end
Standard
Monitoring, Plivo, Redis, Sensu, Sinatra

Phone Call Notification for Sensu Using Plivo

It’s almost 2 weeks since i’ve joined Plivo’s DevOps family. I was spending a lot of time on understanding their API’s as i’m new to telephony. Plivo’s API made telephony so easy that even a person with basic programming language can built powerfull telephony apps to suit to their infrastructure. So when i started playing around with the API, i decided to take it to a different level. I’m strong lover of Sensu Monitoring tool, so i decided to integrate Plivo with Sensu to built a new Notification system using the Phone call. Many people rely on Pagerduty for the same feature. But that part is completly managed by PagerDuty, where in for the alerts which we sent to PagerDuty, they will notify us via phone call to the phone numbers mentioned on Pager Duty. So i decided to built a similar Handler to Sensu, so that we can have a similar feature on Sensu. In this blog i will explain how i achieved the same with Plivo Framework.

Plivo provides Application Programming Interfaces (APIs) to make and receive calls, send SMS, make a conference call, and more. These APIs are used in conjunction with XML responses to control the flow of a call or a message. Developers can create Session Initiation Protocol (SIP) endpoints to perform the telephony operations and the APIs are platform independent and can be used in any programming environment such as PHP, Ruby, Python, etc. It also provides helper libraries for these programming languages.

Here the Sensu will be initiating outbound calls using Plivo’s Call API, to the required numbers and will read out the Alert using Plivo’s Text to Speech Engine. First we need an account on Plivo Cloud, we can go to SignUP page, and need to create an account. By using the default, we can make test calls. But for longer usage, i will suggest to buy some credits for using the service extensively. Once we login with credentials, at the dashboard we can see Plivo AuthID and Plivo AuthToken, which is required to access Plivo’s API. Now for making out bound calls, we need to use the Call API. We also need to provide an ”answer_url” which contains XML instructions to direct the progress of the call. Plivo will fetch the ”answer_url” and executes te XML instructions. So here i will be using Sinatra to create a web app that will returns the XML. The main challenge is the text to be mentioned in the XML need to retrieved from the alert. So we cannot predefine the text as well as the request url, because it will be in dynamic in nature.

Solution :- Here i’m going to use a publically accessible Redis server which is protected with a password. When Sensu handler receives the alert, it will create a hash from the alert received with a random UUID as the name of the hash. And the same UUID will be used as the request path for the answer_url. And when sinatra recieves a request, by default the requested path wont be existing in the sinatra’s config file, as it’s a dynamically generated. So by default Sinatra will return a 404 response. But in Sinatra, there is an option called ”not_found”, where we can customize the response instead of returning 404 directly. So i will be using the “not_found” option and instead of directly returing a 404, i will make sinatra to talk to my redis server. Since the UUID can be fetched from the request URL, and the same is used in the redis as the name of the Hash, Sinatra can get the details of the alert from the Redis. These details are then used to create the XML and will be returned as the response to Plivo. It will be better to go through Plivo’s API documentation to understand more about the XML responses and outbound calls.

For those who don’t have a dedicated server to host the Sinatra app and Redis server, heroku can be used to host the Sinatra app. We can also use the redis addon availabe at the Heroku for our usage.

Sinatra App

Below is the code for the Sinatra app. The ”not_found” option is the one which performs the connection with the Redis.

require 'plivo'
require 'sinatra'
require 'redis'
require 'rest_client'

get '/' do
  play_loop = 1
  lang = "en-US"
  voice = "WOMAN"
  text = "Congratulations! You just made a text to speech app on Plivo cloud!"
  @path = request.path
  puts @path
  speak_params = {
                     'loop' => play_loop,
                     'language' => lang,
                     'voice' => voice,
                     }

  plivo = Plivo::Speak.new(text, speak_params)

  response = Plivo::Response.new()
  response.add(p)
  return response.to_xml
end

not_found do
  @path = request.path
  keyword = @path[1..-1]
  response = Redis.new(:host => "<redis_host_name>", :port => <redis_port>, :password => "<redis_pass>")
  data_red = response.get("#{keyword}")
  if data_red == nil
         return "404 Not Found"
  else
         data = JSON.parse(response.get("#{keyword}"))
         text = data["text"]
         play_loop = 1
         lang = "en-US"
         voice = "WOMAN"
         speak_params = {
                        'loop' => play_loop,
                        'language' => lang,
                        'voice' => voice,
                        }

         plivo = Plivo::Speak.new(text, speak_params)

         response = Plivo::Response.new()
         response.add(p)
         return r.to_xml
   end
end

The above app will redirect all the non found requests and make a requests to the Redis server using the request.path as the name of the hash. If there is a valid hash table, it will generate a Plivo XML else it will return “404 Not Found”.

Plivo Handler

Below is the Plivo Handler code. this is the Initial code, need to be tweaked to suit to the Sensu’s coding standard. Also the settings option need to be added so that all the parameters can be provided in the JSON file of the Handler. But as of now, this handler has been tested with Sensu and it is working perfectly fine.

require 'rubygems'
require 'sensu-handler'
require 'timeout'
require 'securerandom'
require 'plivo'
require 'redis'
require 'timeout'


class PLIVO < Sensu::Handler

  def short_name
    @event['client']['name'] + '/' + @event['check']['name']
  end

  def action_to_string
    @event['action'].eql?('resolve') ? "RESOLVED" : "ALERT"
  end

  def handle
    plivo_auth_id = "XXXXXXXXXXXXXXXX"          # => available at the plivo dashboard
    plivo_auth_token = "XXXXXXXXXXXXXXXX"       # => available at the plivo dashboard
    body = <<-BODY.gsub(/^ {14}/, '')
            #{@event['check']['output']}
            Host: #{@event['client']['name']}
           BODY
    uuid = SecureRandom.hex
    r = Redis.new(:host => "<redis_host_name>", :port => <redis_port>, :password => "<redis_pass>")
    temp = {
            'text' => "#{body}"
            }
    plivo_number = "<YOUR PLIVO NUMBER"
    to_number = "<DESTINATION NUMBER>"
    answer_url = "<YOUR HEROKU APP URL>/#{uuid}"
    call_params = {
                     'from' => plivo_number,
                     'to' => to_number,
                     'answer_url' => answer_url,
                     'answer_method' => 'GET'
                     }
         r.set "#{uuid}", temp.to_json
         r.expire "#{uuid}", <EXPIRY TTL>
         sleep 5
     begin
       timeout 10 do
         puts "Connecting to Plivo Cloud.."
         plivo = Plivo::RestAPI.new(plivo_auth_id, plivo_auth_token)
     details = plivo.make_call(call_params)
       end
       rescue Timeout::Error
       puts "timed out while attempting to contact Plivo Cloud"
         end
  end
end

The above handler will make an outbound call to the Destination number and plivo’s text to speech engine will read out the Alert summary on the call. Now If you want to call multiple user’s, simply create an array or hash with the contact numbers and iterate over it.

This handler will give us the same feature of Phone Call notification similar to Pager Duty. We just need to pay for the phone call usage. Dedicated to all those who wants an affordable yet a good notification system with Sensu. I will be merging out my code soon to the sensu-community repository.

Standard