Elasticsearch, Kibana, logstash, Monitoring, Plivo, SIP, Ubuntu, Voip

Extending ELK Stack to VOIP Infrastructure

Being a DevOps guy, i always love metrics. Visualized metrics gives a good picture of what’s happening in our live battle stations. There are now a quite lot of Open Source tools for monitoring and visualizing. It’s more than a year since i’ve started using Logstash. It never turned me down. ElasticSearch-Logstash-Kibana (ELK) is a killer combination. Though i started Elasticsearch + Logstash as a log analyzer, later StatsD and Graphite took it to the next level. When we have a simple infrastructure it’s easy to monitor. But when the infra starts scaling, it becomes quite difficult to keep track of all the events happening inside each nodes. Though service checks can help, but there is still limitation for it. I faced a lot of scenarios where things breaks but service checks will be fine. Under such scenarios logs are the only hope. They have all these events captured.

At Plivo, we manage a variety of servers from SIP, Media, Proxy, WebServers, DB’s etc. Being a fully Cloud based system, i really wanted to have a system which can keep track of all the live events/status of what’s really happening inside our infra. So my plan was to collect two important stats, 1) Server’s events 2) Application events.

Collectd and Logstash

Collectd is a daemon which collects system performance statistics periodically. Since we have a lot Server’s which handle Realtime Media, it’s a very critical component for us. We need to ensure that the server’s are not getting overloaded and there is no latency in network. I’ve been using Logstash heavily for stashing all my logs. And there is a stable input plugin for collectd to send the all the system metrics to logstash.

First we need to enable the Network Plugin, and then we need to mention our Logstash server IP and port so that collectd can start injecting metrics. Below is a sample colectd configuration.

Hostname    "test.plivo.com"
Interval 10
Timeout 4
Include "/etc/collectd/filters.conf"
Include "/etc/collectd/thresholds.conf"
ReportStats true
    LogLevel info
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin network
<Plugin interface>
    Interface "eth0"
    IgnoreSelected false
<Plugin network>
    Server "{logstash_server_ip}" "logstash_server_port"    # if no port number is mentioned, it will take the default port number (25826)

Now on the Logstash server, we need to add the CollectD plugin on to the input filter in the logstash’s config file.

input {
      collectd {
      port => "5555"    # default port is 25826

Now we are set. Based the plugins enabled in the collectd config file, collctd will start sending the metrics to Logstash on the Interval mentioned in the config, default is 10s. So in my case, i wanted the Load, CPU usage, Memory usage, Bandiwdth (TX and RX) etc. There are default plugins for all these metrics, which we can just enable it in the config file. We also had some custom plugins to collect some custom metrics. BTW writing custom plugin is pretty easy in Collectd.

Now using the Logstash’s Elasticsearch output plugin, we can keep these metrics in Elasticsearch. Now this where Kibana comes in. We can start visualizing these metrics via Kibana. We need to create a custom Lucene Query. Once we have the query, we can create a custom histogram’s for each of these queries. Below aresome sample Lucene queries that we can use with Kibana.

For Load -> collectd_type:"load" AND host:"test.plivo.com"
For Network usage -> collectd_type:"if_octets" AND host:"test.plivo.com"

Below is the screenshot of histogram for Load and Network (TX and RX)

Log Events

Now next is to collect the events from the application logs. We use SIP protocol for all our VOIP sessions. So all our SIP server’s are very critical for us. SIP is pretty similar to HTTP. The response codes are very similar to HTTP responses, ie 1xx, 2xx, 3xx, 4xx, 5xx, 6xx. So i wrote some custom grok patterns so keep track of all of these responses and stores the same on the Elasticsearch.

The second stats which i was interested was our SIP registrar server. We provide SIP endpoints to our customers so that they can use the same with SIP/Soft phones. So i was more interested on stats like Number of registrations/sec, Auth error rates. Plus using ElasticSearch’s MAP facet’s i can create BetterMap. In my previous blog post’s i’ve mentioned on how to create these bettermaps using Kibana and Elasticsearch. Below bettermap screenshot shows us the SIP endpoint registrations from various locations in the last 2 hours.

Now using the Kibana we can start visualizing all these data’s. Below is a sample of Dashboard that i’ve created using Kibana.

ELK stack proved to be an amazing combination. We are currently injecting 3 million events every day and ElasticSearch was blazingly fast in indexing all theses.

kamailio, SIP

Building IM Server using Kamailio

Instant messaging (IM) is a type of online chat which offers real-time text transmission over the Internet. Like XMPP, we can also use SIP for real-time text transmission. The Kamailio PRESENCE module helps us built the same locally. We can convert a simple Kamailio server into a full fledged IM server.

By default, all the registered endpoints will send a “PUBLISH” requests to the Presence server with their status and Presence server stores the status. Also the endpoints will send a “SUBSCRIBE” to other user’s status. For each “SUBSCRIBE” request, the Presence server will sent a “NOTIFY” request back with status of the user. This is how our UserAgent keeps track of each user’s status. Below is simple flow diagram for the same.

Alt text

Setting up Kamailio

In my previous blog, i’ve explained how to install Kamailio from source. The only difference is we need to enable two more modules ”presence” and ”presence_xml”. If Kamailio is already installed, we need to add these two modules into modules.lst at the ”include_modules” line. Once the module names are added, we need to run ”make install” to install the two new modules. Once the module is added, we need to enable the module.

Add “#!define WITH_PRESENCE” into the “kamailio.cfg” file. Then check if there is route logic defined for the “PRESENCE” module. By default there is a PRESENCE route defined in the default “kamailio.cfg” file. If it’s not there below is the route logic. Also we need to add “*route(PRESENCE)” in the ”request_route**” section.

# Presence server route
route[PRESENCE] {

        if(is_method("SUBSCRIBE") && $hdr(Event)=="message-summary") {
                # returns here if no voicemail server is configured
                sl_send_reply("404", "No voicemail service");

        if (!t_newtran())

        } else if(is_method("SUBSCRIBE")) {

        # if presence enabled, this part will not be executed
        if (is_method("PUBLISH") || $rU==$null)
                xlog("@ 404 here 3");
                sl_send_reply("404", "Not here");

Now start the Kamailio server. Now we need to add some users. For that we can use ”kamctl” binary.

$ kamctl add user1@ user1

$ kamctl add user2@ user2

Let’s go ahead and test the settings. For testing we need some IM clients. In my testing, i’ve used Jitsi and adium IM clients. Once the accounts are configured on the clients, it’s better to start a packet capture using wireshark, so that we can see these PUBLISH, SUBSCRIBE and NOTIFY requests between the clients and the Kamailio server. If you are using Adium, go to account options and check the “Publish Status to Everyone” so that it will start sending PUBLISH request to the Kamailio server. Now add the accounts to the Contact lists on the IM clients and then we will be able to see the users status (ONline/OFFline). Now we can start the chat conversation between the user’s.

FreeSwitch, kamailio, SIP

Integrating Kamailio with FreeSwitch

Kamailio aka OpenSER is one of the most powerfull and popular Open Source SIP server. It can be used as SIP Proxy/ Registrar/ LB/ Router etc. It also provides a lot of features like WebSocket support for WebRTC, ; SIMPLE instant messaging and presence with embedded XCAP server and MSRP relay,IMS extensions,ENUM and offcourse AAA (accounting, authentication and authorization) also. Kamailio is a modular system, ie, it has lot of modules which corresponds to particular functions. These modules can be easily installed and can be used easily in Kamailio. In this blog i’m going to use Kamailio as a proxy server. All the user’s are created in the Kamailio and FreeSwitch will be acting as a relay server for outbound calls. So Kamailio performs authentication and all the outbound calls wil be relayed to FreeSwitch.

Installing Kamailio

Lets download the latest version of Kamailio, now it’s 4.1

$ wget http://www.Kamailio.org/pub/Kamailio/latest/src/Kamailio-4.1.1_src.tar.gz

$ tar xvzf Kamailio-4.1.1_src.tar.gz && cd Kamailio-4.1.1

Before we start the build we need to install the basic dependencies.

$ apt-get install gcc flex bison libmysqlclient-dev make libxml2-dev libssl-dev

Now we have the dependencies installed, we can go ahead with the build.

$ make cfg      # generates config files for build system

Now open modules.lst and add the modules to be installed in ”include_modules” section. In my case i’m going to use MySQL backend so it will be ”include_modules= db_mysql” and then we can run the ”make all”. The other way is we can mention the modules directly while running the “make” rather editing the modules.lst file.

$ make include_modules="db_mysql" cfg

Now we can install,

$ make install

The above command will install Kamailio to our system. There are four main binaries for Kamailio,

Kamailio - Kamailio SIP server
kamdbctl - script to create and manage the Databases
kamctl - script to manage and control Kamailio SIP server
kamcmd - CLI - command line tool to interface with Kamailio SIP server

There is also one configuration file called “Kamailio.cfg” which is available by default at /usr/local/etc/Kamailio/Kamailio.cfg

Let’s go ahead with setting up MySQL server for Kamailio.

$ apt-get install mysql-server

Now edit the /usr/local/etc/Kamailio/kamctlrc Locate DBENGINE variable and set it to MYSQL by making ”DBENGINE=MYSQL”. Now we can use the “kamdbctl” binary to craete the default tables and users.

$ /usr/local/sbin/kamdbctl create

The script will add two users in MySQL:

  • Kamailio – (with default password ‘Kamailiorw’) – user which has full access rights to ‘Kamailio’ database

  • Kamailioro – (with default password ‘Kamailioro’) – user which has read-only access rights to ‘Kamailio’ database

There is a sample init.d script available along with Kamailio, which we can use it. We need to copy the sample init.d file to our system’s init.d folder. And the same for the system default file.

$ cp /usr/local/src/Kamailio-4.1.1/pkg/Kamailio/deb/precise/Kamailio.init /etc/init.d/Kamailio

$ cp /usr/local/src/Kamailio-4.1.1/pkg/Kamailio/deb/precise/Kamailio.default /etc/default/Kamailio

$ chmod 755 /etc/init.d/Kamailio 

Edit the new init file and modify the $DAEMON and $CFGFILE values.


$ mkdir -p /var/run/Kamailio    # Directory for the pid file

Default setting is to run Kamailio as user ”Kamailio” and group ”Kamailio”. For that we need to create the user:

$ adduser --quiet --system --group --disabled-password \
      --shell /bin/false --gecos "Kamailio" \
      --home /var/run/Kamailio Kamailio

$ chown Kamailio:Kamailio /var/run/Kamailio

Setting up Kamailio

All the Kamailio configurations are mentioned in only one single file /usr/local/etc/Kamailio/Kamailio.cfg. All the logics are defined in this file, and Kamailio blindly executes this logics and perform the actions. It’s very important that the logics defined in the config should suit to our VOIP platform requirement.

First we need to enable the modules and the necessary features, so add the below lines in the Kamailio.cfg

#!define WITH_MYSQL
#!define WITH_AUTH

We need to define the FreeSwitch server IP and port, for that we can add the below parameters in the “Custom Parameters” section.

freeswitch.bindip = "" desc "FreeSWITCH IP Address"
freeswitch.bindport = "5090" desc "FreeSWITCH Port"

Now we can go ahead to the ”request_route” section which performs the routing logic. Here i’m going to add two more routing logic for the FreeSwitch relay. After the ”request_route” section, we can see the definition for each routing options. Below that we need to add our new route definitions.

        # dial number selection

route[FSRELAY] {
        $du = "sip:" + $sel(cfg_get.freeswitch.bindip) + ":" + $sel(cfg_get.freeswitch.bindport);
                $var(newbranch) = 0;

By default, all the routes mentioned in the “request_route” will be executed line by line. There is a default route called ”Location”, which splits the user part from the request URI and verifies if the user exists in the location table. But when we dial an outside number/user, this location check will fail, so i’m going to add a condition which checks if the user in the request URI contains a number with a length 9-15 will be relayed to the FreeSwitch. Again this is just a simple condition, we can create a more complex condition, like check the domain part, if the domain part contains a domain which doesnot belong to our Domain list, we can either decline the request, or we can relay to FreeSwitch or we can make DNS query and we can make Kamailio to process the request to that domain’s Proxy server. Like this we can define our own conditions in the config file, and Kamailio will execute it line by line.

I’m going to add my check condition on the ”LOCATION” route definition.

route[LOCATION] {

    #!ifdef WITH_SPEEDDIAL
        # search for short dialing - 2-digit extension
        if($rU=~"^[0-9]{9,15}$")        # checking for numbers in the Request URI
    #!ifdef WITH_ALIASDB
        # search in DB-based aliases
        $avp(oexten) = $rU;
        if (!lookup("location")) {
                xlog("L_INFO", "CALL $rm $ci lookup here\n");
                xlog("L_INFO", "$fU@$fd - Lookup contact location for $rm\n");
                xlog("L_INFO", "rc is $var(rc)");
                switch ($rc) {
                        case -1:
                        case -3:
                                xlog("L_ERR", "$fU@$fd - No contact found\n");
                                send_reply("404", "Not Found here");
                        case -2:
                                send_reply("405", "Method Not Allowed");

        # when routing via usrloc, log the missed calls also
        if (is_method("INVITE"))
        xlog("L_INFO", "CALL $rm $ci relay\n");
        xlog("L_INFO", "$fU@$fd - Relaying $rm\n");

So now all the calls coming with numbers of length 9-15 in the Request URI will be relayed to the FreeSwitch, and FreeSwitch will process the call based on the DialPlan configured in the FreeSwitch. Since i’m going to use IP authentication, i need to whitelist the Kamailio ip in “acl_conf.xml” file in the FreeSwitch autload conf directory, so that FreeSwitch will accept the invites from Kamailio. Again i’m not defining any Voicemail options here. If we have a Voice mail server, then we can create another route option and when the caller doesn’t pick the call we can route the call to the Voice Mail server.
For example the below condition will route the failures to FreeSwitch Voice Mailbox.

            # in case of failure - re-route to FreeSWITCH VoiceMail
            t_on_failure("FAIL_FSVBOX");        # where FSVBOX is a route definition

Kamailio has a lot of modules which really comes in handy. For example we can use LDAP module to use LDAP as a backend. There is a PRESENCE module which helps us to setup an Instant Messaging server using Kamailio. I’ll be writing a blog on how to use Kamailio as an IM server soon. One of the main advantage of Kamailio over OpenSIP is the WebSocket support. This just a basic configuration, but we can design much complex system using Kamailio. We can even remove the default route logics, and we can use our own routing logics. Kamailio doesnot depend on default routing logics, it blindly reads the route and executes it for each incoming connections.