Managing Docker Clusters Using Mesos and Marathon

Docker has became one of my favourite tool. It’s super cool and super easy tool to manage linux containers. LXC’s are around in IT world for some time, but by the entry of Docker last year, the wave started rising. Thanks to Docker team and Solomon Hykes for open sourcing such a wonderfull project. I’ve already mentioned a lot of stuffs about Docker in my previos blogs, so today im going explain how Docker can be used as a Cluster. There are some interesting tools like CoreOS, Helios etc for managing Docker as a cluster. But today i’m going to explain on how to set up a Docker cluster using Apache Mesos. CoreOS is a custom linux os which comes with SystemD. But the restriction is, we have to use that custom images of coreos.. Indeed CoreOS team open sourced some exciting tools like etcd fleet which works with CoreOS for managing Docker clusters. But Mesos is quite simple, we can install it via package, or even using tar balls available in thier Github repo onto most of the Linux Distro’s and it’s quite easy to configure also. Mesos is heavily used by Twitter to manage their data center’s. And now Mesosphere has opensourced a new tool called Mararthon which now provides a UI and a Rest API for maaging and scheduling Mesos Frameworks aka jobs, in this case containers as a service.

A few weeks ago, Mesos 0.19 was released which comes with an official support for Docker coantiners by intergrating Deimos into it. And a few days ago Marathon has released their new version 0.6.0 supports launching any task in a Docker container via Mesos 0.19+

Setting up Mesos Cluster

In this test setup, i’m going to setup both Mesos master/slave and Zookeeper on the same Ubuntu 14.04 vagrant node. First we can install the dependencies,

$ apt-get install curl python-setuptools python-pip python-dev python-protobuf

Now we can install Zookeeper

$ apt-get install zookeeperd

After the installation, ZooKeeper has 1 configuration. Each Zookeeper needs to know its position in the quorum.

$ echo 1 | sudo dd of=/var/lib/zookeeper/myid

Now we can setup Docker

$ echo "deb http://get.docker.io/ubuntu docker main" > /etc/apt/sources.list.d/docker.list

$ apt-get update && apt-get install lxc-docker

$ docker version

   Client version: 1.0.0
   Client API version: 1.12
   Go version (client): go1.2.1
   Git commit (client): 63fe64c
   Server version: 1.0.0
   Server API version: 1.12
   Go version (server): go1.2.1
   Git commit (server): 63fe64c

Let’s pull some basic ubuntu images from Docker Hub so that we can use the same for testing.

$ docker pull libmesos/ubuntu

Now we can configure Mesos

$ curl -fL http://downloads.mesosphere.io/master/ubuntu/14.04/mesos_0.19.0~ubuntu14.04%2B1_amd64.deb -o /tmp/mesos.deb

$ dpkg -i /tmp/mesos.deb

$ mkdir -p /etc/mesos-master

$ echo in_memory | sudo dd of=/etc/mesos-master/registry

## Mesos Python egg for use in authoring frameworks

$ curl -fL http://downloads.mesosphere.io/master/ubuntu/14.04/mesos-0.19.0_rc2-py2.7-linux-x86_64.egg -o /tmp/mesos.egg

$ easy_install /tmp/mesos.egg

We can download the latest Marathon 0.6 from here

$ tar xvzf marathon-0.6.0.tgz

Mesos uses Deimos for managing dockers, Deimos can installed via pip

$ pip install deimos

Also, we need to configure mesos to use Deimos,

$ mkdir -p /etc/mesos-slave

$ echo /usr/local/bin/deimos | sudo dd of=/etc/mesos-slave/containerizer_path

$ echo external | sudo dd of=/etc/mesos-slave/isolation

Now we can start all the services.

$ initctl reload-configuration

$ service docker start

$ service zookeeper start

$ service mesos-master start

$ service mesos-slave start

##### Starting Marathon #####

$ cd marathon-0.6.0

$ ./bin/start --master zk://localhost:2181/mesos --zk_hosts localhost:2181

Marathon will now start listening to port 8080, We can access the UI from the browser via this port, also via rest API using the same port.

curl localhost:8080/help   # gives us some details about the API's

I just went through the Deimos code, so under the hood they are using docker run with some default parameters like --sig-proxy, --rm, --cidfile, -v, -w and extra parameters that we are passing while creating the task via Marathon.

As of now, we still can’t pass details like Container image, Docker options via Marathon GUI. So we can use the Rest API for the time being. Below is a sample curl request for launcing a single container,

curl -X POST -H "Accept: application/json" -H "Content-Type: application/json" \
    localhost:8080/v2/apps -d '{
        "container": {"image": "docker:///libmesos/ubuntu", "options": ["--privileged"]},
        "cpus": 0.5,
        "cmd": "sleep 500",
        "id": "docker-tester",
        "instances": 1,
        "mem": 300
    }'

We can pass custom options to the docker run command via “options”. After making the curl request, we can check the syslog, as mesos will be logging into syslog by default. We can even see the Docker run command on the same.

Jun 27 07:24:58 vagrant-ubuntu-trusty-64 deimos[19227]: deimos.containerizer.docker.launch() exit 0 // docker run --sig-proxy --rm --cidfile /tmp/deimos/mesos/00d459fb-22ca-4af7-9a97-ef8a510905f2/cid -w /tmp/mesos-sandbox -v /tmp/deimos/mesos/00d459fb-22ca-4af7-9a97-ef8a510905f2/fs:/tmp/mesos-sandbox --privileged -p 31498:31498 -c 512 -m 300m -e PORT=31498 -e PORT0=31498 -e PORTS=31498 libmesos/ubuntu sh -c 'sleep 500'

We can also use the Marathon Rest API to check the status of the job which we started.

curl -X GET -H "Content-Type: application/json" localhost:8080/v2/apps

Below is the screenshort for the same from the Marathon UI.

We can also check if the container is launched via docker ps command.

A more detailed report about the Docker job which we have launched can be viewed via the default Mesos GUI listening on port 5050 on the Mesos master. Now we can test the scalability of the Job. Currently we have only one container running. So now we can try scaling say adding one more node. We can do it in two ways, like via PUT request using curl or using GUI

curl -X PUT -H "Content-Type: application/json" localhost:8080/v2/apps/docker-tester \
    "container": {"image": "docker:///libmesos/ubuntu", "options": ["--privileged"]},
            "cpus": 0.5,
            "cmd": "sleep 500",
            "id": "docker-tester",
            "instances": 2,     # increasing the instance count to 2
            "mem": 300
            }'

Now we can use the docker ps command to see if the new container is launched or not. Also we can see that status in UI also.

Similarly, we can scale down also. I’ve tested the same and all seems to be good. Marathon ensures that the docker process will be running. So incase if the process crashes Marathon will restart the same and ensures that the instances are up and running as per our configuration. There are a few other Open Sourced Mesos Scheduler’s like Apache Aurora, Airbnb’s Chronos. But for my requirement marathon is pretty straight and simple and also provides a very good Rest API layer for managing containers. Mesos, Marathon and Docker are still young, but provides a killer combination for managing clusters built over Docker containers.

Dockerizing FreeSwitch – Docker Enters Telephony World

Docker has became one of the hottest topics in IT now a days. Docker is an open-source project that automates the deployment of applications inside software containers. Docker extends a common container format called Linux Containers (LXC), with a high-level API providing lightweight virtualization that runs processes in isolation.Docker uses LXC, cgroups, and the Linux kernel itself. Though i coudn’t make out to the DockerCon 2014 in SF, a lot new developments were announced on the DockerCon. Especially three new Opensource Projects libcontainer, libchan and libswarn. Docker is indeed creating a revolution in the container space, creating a next generation of scalable platform management. There are a lot PAAS services like Deis, resin.io, Dokku which are already using Docker in production. Another important and exciting project is CoreOS. CoreOS uses tools like SystemD, Fleet, EtcD to build a fully scalabale docker based cluster management system. I definitely need a separate blog to write about CoreOS, it’s really a super exciting project to play with.

Last week Docker Team released Version 1.0 of Docker. So i’ll be using the same in this new set up. It’s been almost 6 Month’s since i’ve been working @ Plivo as a DevOps Engineer. Telephony was really a very new platform for me. And my first companion was offcourse FreeSwitch,a scalable open source cross-platform telephony platform designed to route and interconnect popular communication protocols using audio, video, text or any other form of media. I was heavily using Vagrant for all my experiments in my mac. But after started using Docker, it really made me crazy. I’ve played for some time wiht LXC’s long back. So this was like a leap back to the container world.

There are a lot of concerns on using Virtual Machines in Telephony world. Especially for the server’s that handles the Real Time voice packets, as voice quality is pretty important in Telephony. Docker’s again more light weight isolated environment, and i decide to see how Docker can perform with such issues. If Docker handle Freeswitch smoothly, then i’m sure that we can use Docker for other telephony app’s like OpenSIPS/Kamailio etc, as they handle only sessions not the Media traffic. I know there are a lot of concerns like CPU load, Network etc, but this is like an initial move to test Docker into Telephony.

Setting Up Docker

Docker 1.0 is available from the Official Docker repo.

$ echo "deb http://get.docker.io/ubuntu docker main" > /etc/apt/sources.list.d/docker.list

$ apt-get update && apt-get install lxc-docker

Now we can check the Docker version using the docker binary itself.

$ docker version

Client version: 1.0.0
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 63fe64c
Server version: 1.0.0
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 63fe64c

Now Docker is installed, but we need some OS images to use with docker. We can build custom images using debootstrap etc. But there are official minimal images available in Docker HUB. We can search for the repositories and can pull those images via docker binary itself.

For example to pull the entire Ubuntu images, we can just do,

$ docker pull ubuntu

But this will download all the ubuntu images available in the repo. We can also do selective download by using the tag.

$ docker pull ubuntu:14:04

Once the images are downloaded, we can use images option in docker binary to see all the downloaded images.

$ docker images

REPOSITORY                      TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
ubuntu                          14.04               ad892dd21d60        10 days ago         275.5 MB

Here i’m not going to daemonize the container, i’ll be using the interactive option. But first, let’s start a new container.

$ docker run -t -i ubuntu:14.04 /bin/bash

This command will start a conatiner and will open up a bash session for us and we will be inside the bash session. Now to use an application we need to open up corresponding ports to outside world. We can use the “-p” option while starting a docker container to enable port forwarding. Under the hood, docker is using IPtables for the same. In the case of Freeswitch, we need to open 5060,5080 for the default Sofia profiles (Internal and External). Also we need to open the RTP ports. In this test i’ll be opening a predefined set of ports ie from “16384” to “16394”. (As my Docker host resides on Azure, creating an Endpoint for each port forward is really a pain, so i decided to open only a few). And also i’ll be opening port 22, so that we can have an ssh server inside the container.

$ docker run -t -i -p 2223:22 -p 5060:5060/tcp -p 5060:5060/udp -p 16384:16384/udp -p 16385:16385/udp -p 16386:16386/udp -p 16387:16387/udp -p 16388:16388/udp -p 16389:16389/udp -p 16390:16390/udp -p 16391:16391/udp -p 16392:16392/udp -p 16393:16393/udp -p 5080:5080/tcp -p 5080:5080/udp ubuntu:14.04 /bin/bash

This will start a new container and Docker by default will setup the IPtables for port forwarding. So now my IPtables looks like this.

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   43 16850 ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:5080
    0     0 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           tcp dpt:5080
  988  198K ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:16392
    0     0 ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:16389
    0     0 ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:16385
    0     0 ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:16393
 2026  405K ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:16388
 8817 1763K ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:16384
12144 8684K ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:5060
 4359  257K ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           tcp dpt:5060
 9917 1983K ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:16390
    0     0 ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:16387
    0     0 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           tcp dpt:22
   38  4848 ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:16391
    1   152 ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.6           udp dpt:16386
    0     0 ACCEPT     all  --  *      lxcbr0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  lxcbr0 *       0.0.0.0/0            0.0.0.0/0
 431K  630M ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
 128K   19M ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
   16  2460 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

Now we can go ahead with Freeswitch compilation. In my previous blog, i’ve mentioned how to compile and set up freeswitch. Once freeswitch is ready, we need to make a few changes. By default, Freeswitch uses STUN to route through NAT, but this doesn’t work with Docker. So we have to set the external IP manually. In the Freeswitch installed folder, edit conf/autoload_configs/switch.conf.xml. In this file we can set the External IP manually. Add the below lines to switch_conf.xml.

<X-PRE-PROCESS cmd="set" data="external_sip_ip=<YOUR_EXTERNAL_IP>"/>
<X-PRE-PROCESS cmd="set" data="external_rtp_ip=<YOUR_EXTERNAL_IP>"/>

Also we need to modify the Default Sofia Profiles and need to set the ext-rtp-ip and ext-sip-ip to use our external IP added in the switch_conf.xml file while establishing connections. Add the below lines to the conf/sip_profiles/internal.xml and conf/sip_profiles/external.xml

<param name="ext-rtp-ip" value="$${external_rtp_ip}"/>
<param name="ext-sip-ip" value="$${external_sip_ip}"/>

Now we need to set teh RTP ip range to the range which we have forwarded while creting the container. So we need to edit conf/autoload_configs/switch.conf.xml

<param name="rtp-start-port" value="16384"/>
<param name="rtp-end-port" value="16394"/>

Once the changes are made, we can start the FreeSwitch service. Now to make sure that the External IP is working properly, we can check the sofia profile status using fs_cli. below is a sample output of the sofia profile status.

freeswitch@internal> sofia status profile internal
=================================================================================================
Name                internal
Domain Name         N/A
Auto-NAT            false
DBName              sofia_reg_internal
Pres Hosts          172.17.0.6,172.17.0.6
Dialplan            XML
Context             public
Challenge Realm     auto_from
RTP-IP              172.17.0.6
Ext-RTP-IP          <my_external_ip>
SIP-IP              172.17.0.6
Ext-SIP-IP          <my_external_ip>
URL                 sip:mod_sofia@<my_external_ip>:5060
BIND-URL            sip:mod_sofia@<my_external_ip>:5060;maddr=172.17.0.6;transport=udp,tcp
HOLD-MUSIC          local_stream://moh
OUTBOUND-PROXY      N/A
CODECS IN           OPUS,G722,PCMU,PCMA,GSM
CODECS OUT          OPUS,G722,PCMU,PCMA,GSM
TEL-EVENT           101
DTMF-MODE           rfc2833
CNG                 13
SESSION-TO          0
MAX-DIALOG          0
NOMEDIA             false
LATE-NEG            true
PROXY-MEDIA         false
ZRTP-PASSTHRU       true
AGGRESSIVENAT       false
CALLS-IN            0
FAILED-CALLS-IN     0
CALLS-OUT           0
FAILED-CALLS-OUT    0
REGISTRATIONS       1

Now freeswitch ahs started successfully. We can test some basic calls using softphones like Xlite, Telephone etc. By default, there are some default extensions and user’s available, so we can use the same for testing the calls. But i really wanted to try trunkning also and wanted to see the quality of the voice. So i created SIP trunking in Freeswitch using Plivo. And i tested a couple of calls to US and India DID’s and no issues were detected in the quality. But again i need to test the laod of the server’s when it startes handling concurrent calls and also the voice quality. But i decied to d oit as a Phase II. But as of now, Docker FreeSwitch is working perfectly like a physical machine with out issues.

So now we have a working FreeSwitch container, now here comes the main advantage of the Docker. We can create a new image with all these changes, so that nex time i dont need to work from scratch. I can use this saved image and a readymade Docker Freeswitch container can be launched in seconds. Since we are in interactive mode, we should not quit the session before it’s saved or else all the things will be lost,becoz dokcer will destroy the same. So open up a new shell on the docker host and use the commit option. But to use the commit command, we need to know the container id, so here docker ps command comes handy.

$ docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS           NAMES

e7f3c02346d4        a4196763d248        /bin/bash           32 hours ago        Up 32 hours         0.0.0.0:2223->22/tcp, 0.0.0.0:5060->5060/tcp, 0.0.0.0:5060->5060/udp, 0.0.0.0:5080->5080/tcp, 0.0.0.0:5080->5080/udp, 0.0.0.0:16384->16384/udp, 0.0.0.0:16385->16385/udp, 0.0.0.0:16386->16386/udp, 0.0.0.0:16387->16387/udp, 0.0.0.0:16388->16388/udp, 0.0.0.0:16389->16389/udp, 0.0.0.0:16390->16390/udp, 0.0.0.0:16391->16391/udp, 0.0.0.0:16392->16392/udp, 0.0.0.0:16393->16393/udp   silly_turing

In my case “e7f3c02346d4” is the container ID. So i can use the same for commit. I won’t be commiting to the base Ubuntu image, as i can use the same for other purposes, so here i’ll commiting to a new image say “ubntu-fs-docker”

$ docker commit -m "<commit message>" e7f3c02346d4 ubntu-fs-docker

Now we can use this “ubntu-fs-docker” image to launch a ready made FreeSwitch server’s.

Docker is a very juvenile project about more than a year old. But the use cases are expanding heavily in the Modern IT world. Docker is fueling up a new generation of scalable servers. Wishing all the best for Docker and kudos to Solomon Hykes and the DotCloud team for opensourcing such a powerfull project

UCHIWA – Awesome Dashboard for Sensu

It’s been more than a year since i’ve started playing with Sensu. It was one of coolest monitoring projects that i’ve ever worked with. Perfect for Cloud infrastructure and backed by a cool community. Though Sensu is still in Juvenile (v 0.12) state, its mature enough to tackle majority of the monitoring issues. Recently i started migrating all our monitoring from traditional Nagios to Sensu and the pretty cool thing with sensu is, we can directly use the Nagios plugins with Sensu. That’s an easy task for migration. We don’t have to rebuild all the check’s to make with Sensu. But most of the people outside was pointing to the sensu’s default dashboard. Though the dashboard doesn’t looks pretty fancy, it can do all the functions. But having a good dashboard which can display the current status is always a time saver.

So i was being searching for a good dashboard and the first choice on the google search was Sensu-Admin. A Rails project, which needs a backend DB. But still i was not satisfied with it, and i started looking out for something different. The second choice was sabisu. Sabisu uses Cloudant’s hosted Couchdb with Lucene. We just need to store all the events in a Redis List and a custom script which reads the data from the Redis List and pushes it to the Cloudant’s CouchDB. So we basically need a Cloudant account and the Webapp makes Lucene queries to the Cloudant DB and displays the results on the Sabisu dashboard. Though i tried to rebuild the same setup locally, like running a CouchDB+Lucene locally and sending the same data to the local couchdb. With some codehack’s i was able to make the webapp talk to my local CouchDB and display the results on the dashboard.

But then, I found a super cool dashboard project called UCHIWA which was started in Github a few days back by Simon Plourde. Uchiwa is simple dashboard built with NodeJS and uses SocketIO for real time updates. The screenshot’s looks super cool and i decided to give it a try. It has only one dependency, NodeJS, no backend DB’s required as it talks to the Sensu’s API in realtime.

Setting Up NodeJS

For Ubuntu, we can use the chris-lea’s PPA.

apt-get install python-software-properties        # required for "apt-add-repository" binary

apt-add-repository ppa:chris-lea/node.js

apt-get update

apt-get install nodejs

now we have the latest NodeJS on our system, we can start setting up Uchiwa.

Setting Up Uchiwa Dashboard

Uchiwa’s source is available in Github.

git clone https://github.com/palourde/uchiwa.git

Once cloned, the repository contains the “package.json” file which contains the list of necessary dependencies. We can use “npm” (node package manager) to install all these.

cd uchiwa

npm install

Now we need to create a config file for the app. There is a sample config file available in the repo. So we need to mention the Sensu’s API IP and Port number and also the auth credentials if any. Plus auth credentials for accessing Uchiwa Dashboard page.

once all these are set, we are done. We just need to start the service.

node app.js 

We can access the page via http://localhost:3000/, or we can proxy pass from the webserver. Instructions for Nginx is available on the Readme of the project. Now we need to keep this app running all the time. So it’s better to create a init/upstart process for the same, so that the process will start automatically when the system reboots. There is a cool Node project called forever which is a simple CLI tool for ensuring that a given script runs continuously.

I’ve created an upstart script for Uchiwa, bu putting a conf file “uchiwa.conf” in the “/etc/init” directory. Below is the content for the conf file. Once the file is in place, we have to do a reload of the upstart configuration. initctl reload-configuration will do the trick.

description "uchiwa - dashboard for sensu"
env APP_PATH = "/usr/local/uchiwa/"

start on startup
stop on shutdown

script
  cd $APP_PATH
  exec forever start app.js
end script

Uchiwa looks pretty cool and neat and it has the stash support also. There are couple of addons required like “Downtime”, but Uchiwa is a pretty new project and i’m sure that this project is gonna grow soon. It has already received 99 stars on the Github. Kudos to Simon Plourde for Open Sourcing this awesome project.

Extending ELK Stack to VOIP Infrastructure

Being a DevOps guy, i always love metrics. Visualized metrics gives a good picture of what’s happening in our live battle stations. There are now a quite lot of Open Source tools for monitoring and visualizing. It’s more than a year since i’ve started using Logstash. It never turned me down. ElasticSearch-Logstash-Kibana (ELK) is a killer combination. Though i started Elasticsearch + Logstash as a log analyzer, later StatsD and Graphite took it to the next level. When we have a simple infrastructure it’s easy to monitor. But when the infra starts scaling, it becomes quite difficult to keep track of all the events happening inside each nodes. Though service checks can help, but there is still limitation for it. I faced a lot of scenarios where things breaks but service checks will be fine. Under such scenarios logs are the only hope. They have all these events captured.

At Plivo, we manage a variety of servers from SIP, Media, Proxy, WebServers, DB’s etc. Being a fully Cloud based system, i really wanted to have a system which can keep track of all the live events/status of what’s really happening inside our infra. So my plan was to collect two important stats, 1) Server’s events 2) Application events.

Collectd and Logstash

Collectd is a daemon which collects system performance statistics periodically. Since we have a lot Server’s which handle Realtime Media, it’s a very critical component for us. We need to ensure that the server’s are not getting overloaded and there is no latency in network. I’ve been using Logstash heavily for stashing all my logs. And there is a stable input plugin for collectd to send the all the system metrics to logstash.

First we need to enable the Network Plugin, and then we need to mention our Logstash server IP and port so that collectd can start injecting metrics. Below is a sample colectd configuration.

Hostname    "test.plivo.com"
Interval 10
Timeout 4
Include "/etc/collectd/filters.conf"
Include "/etc/collectd/thresholds.conf"
ReportStats true
    LogLevel info
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin network
<Plugin interface>
    Interface "eth0"
    IgnoreSelected false
</Plugin>
<Plugin network>
    Server "{logstash_server_ip}" "logstash_server_port"    # if no port number is mentioned, it will take the default port number (25826)
</Plugin>

Now on the Logstash server, we need to add the CollectD plugin on to the input filter in the logstash’s config file.

input {
      collectd {
      port => "5555"    # default port is 25826
      }
}

Now we are set. Based the plugins enabled in the collectd config file, collctd will start sending the metrics to Logstash on the Interval mentioned in the config, default is 10s. So in my case, i wanted the Load, CPU usage, Memory usage, Bandiwdth (TX and RX) etc. There are default plugins for all these metrics, which we can just enable it in the config file. We also had some custom plugins to collect some custom metrics. BTW writing custom plugin is pretty easy in Collectd.

Now using the Logstash’s Elasticsearch output plugin, we can keep these metrics in Elasticsearch. Now this where Kibana comes in. We can start visualizing these metrics via Kibana. We need to create a custom Lucene Query. Once we have the query, we can create a custom histogram’s for each of these queries. Below aresome sample Lucene queries that we can use with Kibana.

For Load -> collectd_type:"load" AND host:"test.plivo.com"
For Network usage -> collectd_type:"if_octets" AND host:"test.plivo.com"

Below is the screenshot of histogram for Load and Network (TX and RX)

Log Events

Now next is to collect the events from the application logs. We use SIP protocol for all our VOIP sessions. So all our SIP server’s are very critical for us. SIP is pretty similar to HTTP. The response codes are very similar to HTTP responses, ie 1xx, 2xx, 3xx, 4xx, 5xx, 6xx. So i wrote some custom grok patterns so keep track of all of these responses and stores the same on the Elasticsearch.

The second stats which i was interested was our SIP registrar server. We provide SIP endpoints to our customers so that they can use the same with SIP/Soft phones. So i was more interested on stats like Number of registrations/sec, Auth error rates. Plus using ElasticSearch’s MAP facet’s i can create BetterMap. In my previous blog post’s i’ve mentioned on how to create these bettermaps using Kibana and Elasticsearch. Below bettermap screenshot shows us the SIP endpoint registrations from various locations in the last 2 hours.

Now using the Kibana we can start visualizing all these data’s. Below is a sample of Dashboard that i’ve created using Kibana.

ELK stack proved to be an amazing combination. We are currently injecting 3 million events every day and ElasticSearch was blazingly fast in indexing all theses.

Near RealTime Dashboard with Kibana and Elasticsearch

Being in DevOps it’s always Multi tasking. From Regular customer queries it goes through Monitoring, Troubleshooting etc. And offcourse when things breaks, it really becomes core multi tasking. Especially when you have a really scaling infrastructure, we should really understand what’s really happening in our infrastructure. Yes we do have many new generation cloud monitoring tools like Sensu, but what if we have a near real time system that can tell us the each and every events happeing in our infrastructure. Logs are the best places where we can keep track of the events, even if the monitoring tools has missed it. We have a lot of log aggregator tools like tool Logstash, Splunk, Apache Kafka etc. And for log based event collection the common choice will be always Logstash -> StatsD -> Graphite. And ElasticSearch for indexing these.

My requirement was pretty straight. Record the events, aggregate them and keeps track of them in a timely manner. Kibana uses ElasticSearch facets for aggregating the search query results.Facets provide aggregated data based on a search query. So as a first task, i decided to visualize the location of user’s who are registering their SIP endpoints on our SIP registrar server. Kibana gives us a good interface for the 2D heat map as well as a new option called BetterMap. Bettermap uses geographic coordinates to create clusters of markers on map and shade them orange, yellow and green depending on the density of the cluster. So from the logs, i just extracted the register events, and used a custom regex patterns to extract the details like the Source IP, usernames etc using logstash. Using the logstash’s GeoIP filter, the Geo Locations of the IP can be identified. For the BetterMap, we need coordinates, in geojson format. GeoJSON is [longitude,latitude] in an array. From the Geo Locations that we have identified in the GeoIP filter, we can create this GeoJSON for each event that we are receiving. Below is a sample code that i’ve used in logstash.conf for creating the GeoJSON in Logstash.

if [source_ip]  {
    geoip {
      type => "kamailio-registers"
      source => "source_ip"
      target => "geoip"
      add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
      add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
  }

The above filter will create a GeoJSON array “geoip.coordinates”. This array can be used for creating the BetterMap in Kibana. Below are the settings for creating a BetterMap panel in the Kibana dashboard. While adding a new panel, select “bettermap” as the panel type, and the co-ordinate filed should be the one which contains the GeoJSON data. Make sure that the data is of the format [longitude,latitude], ie Longitude first and then followed by latitude.

Moving ahead, i decided to collect the events happening on our various other server’s. We were one of the earliest companies who started using SIP (Session Initiation Protocol). SIP employs design elements similar to the HTTP request/response transaction model. So similar to web traffic, i’ve decided to collect events related to 4XX, 5XX and 6XX error responses, as it is very important to us. Once the logs are shipped to logstash, i wrote another custom grok pattern, which extracts the Error Code and Error responses, including the server which returned the same. These data’s can be used for future analysis also. So i decided to store these on ElasticSearch. So now we have the real time event data’s stored, but how to visualize. Since i dont have to perform much mathematical analytics with data, i decided to to remove graphite. Kibana has a wonder full GUI for visualizing the data. So decided to go ahead with Kibana. One option is “histogram” panel time. Using histogram we can visualize the data via a regular bar graph, as well as using the area graph. There is another panel type called “terms” which can be used to display the agrregated events via pie chart, bar chart, or a table. And below is what i achieved with Kibana.

This is just an inital setup. I’m going to add more events to this. As of now Kibana + Elasticsearch proves to be a promising combination for displaying all near real time events happening in my Infrastructure.

Event Monitoring Using Logstash + StatsD + Riemann

Being an OPS guy, i love Logs a lot. Logs contains lots of sensitive events recorded in it. Though a lot of people rely on monitoring tools, there are a lot of scenario where we still can’t rely on monitoring. In such scenarios, logs are the best sources to identify those events in a near real time fashion. A common scenario is Web Operations, where we need to count the the various 4xx, 5xx, Auth errors experienced by the user’s. I had a simliar requirement where i need to identify the 4xx, 5xx, 6xx errors and other similar failures on various SIP server’s. But apart from just visualising these error’s i also wanted a notification system which can notify me when the value crosses the threshold.

Logstash and StasD is a perfect combination for aggregating events from the logs. StatsD has a Graphite backend, where it sends the aggreagated metric values for visualizing. But when we have large number graphs, and offcourse when being a multi tasking Ops guy, it’s not possible to sit and watch all these graphs. So we need a notification system which alert’s us when things starts breaking. Here comes RIEMANN. Riemann aggregates events from your servers and applications with a powerful stream processing language. Riemann is pretty light weight, easy to configure monitoring framework. Logstash sents the filtered events from the logs to StatsD output plugin. Based on the flushInterval, statsD iterates through the received events sents the aggregated metric values to the Graphite. There is also a Riemann output plugin for Logstash, but we need to pass the state/metric to the plugin. In my case, logstash filters the event from the log, so i need to converts these events to time based metric values. Since statsD already has these events converted into time series metrics, i decided to write a small backend for statsD that can send these aggregated metrics to Riemann.

The StatsD backend basically requires to main functions, one is ”flush_stats” which will get invoked once the flush interval is reached. This function then iterates over the received metrics and passes these aggregated metrics to another function called ”post_stats”, which sends the metrics to the corresponding aplications. In our case, we need to send the metrics to Riemann. There is a Riemann-Node plugin, which we can utilize here for sending the metrics to Riemann server. Below is the content for the ”flush_stats” and ”post_stats” functions. Currently i’ve added support only for counters. Soo i’ll be adding support for Counters and Timers also.

flush_stats function
--------------------

var flush_stats = function riemann_flush(ts, metrics) {
var statString = '';
var numStats = 0;
var key;

var counters = metrics.counters;
var gauges = metrics.gauges;
var timers = metrics.timers;
var pctThreshold = metrics.pctThreshold;

for (key in counters) {
    var value = counters[key];
    var valuePerSecond = value / (flushInterval / 1000); // calculate "per second" rate

    statsString = value;
    service = key;
    time_stamp = ts;
    post_stats(statString, service_name, time_stamp);
}
}; 


post_stats function
-------------------

var post_stats = function riemann_post_metrics(statString, service_name, time_stamp) {

riemannStats.last_exception = Math.round(new Date().getTime() / 1000);

client.send(client.Event({
  service: service_name,
  metric:  statsString,
  time: time_stamp
}));
};

So here i’m not gonna send the per second metrics. I’m using the default 10 sec flushInterval. So every seconds StatsD will send the incremented metrics to Riemann. The namespace, sender etc are defined in the logstash conf itself. The full plugin file is available in here

To use this Riemann backend, first we need to copy this file into the backend folder of the StatsD repo folder. Then we need to enable this plugin in the StatsD config file. Below is a sample config file which uses both graphite and Riemann backends.

{
  riemannPort: 5555
, riemannHost: "localhost"
, graphitePort: 2003
, graphiteHost: "localhost"
, port: 8125
, backends: [ "./backends/riemann", "./backends/graphite" ]
}

So now StatsD will send out the incremented and the per second metric to Graphite and the Riemann backend will send the incremented metric to the Rieman server. No we can define the metric threshold and the notification method on the reimann config file. Below is my reimann metric threshold and notification.

(streams
      (where (>= metric 10)
        (where (service #"SIP")
          (email "deepakmdass88@gmail.com")))))

So whenever the recived metric value is beyond 10, Riemann will notify the same to my Email. I’ve done some dry testing with this setup. So far this setup never turned me down. Though there are some tweaks to be done, but this setup really suited to my requirement. being an OPS guy, my primary focus is to detect the outages at a very early stages to minimize the impact. Hope this guy will be an added defence layer for the same.

Building IM Server using Kamailio

Instant messaging (IM) is a type of online chat which offers real-time text transmission over the Internet. Like XMPP, we can also use SIP for real-time text transmission. The Kamailio PRESENCE module helps us built the same locally. We can convert a simple Kamailio server into a full fledged IM server.

By default, all the registered endpoints will send a “PUBLISH” requests to the Presence server with their status and Presence server stores the status. Also the endpoints will send a “SUBSCRIBE” to other user’s status. For each “SUBSCRIBE” request, the Presence server will sent a “NOTIFY” request back with status of the user. This is how our UserAgent keeps track of each user’s status. Below is simple flow diagram for the same.

Alt text

Setting up Kamailio

In my previous blog, i’ve explained how to install Kamailio from source. The only difference is we need to enable two more modules ”presence” and ”presence_xml”. If Kamailio is already installed, we need to add these two modules into modules.lst at the ”include_modules” line. Once the module names are added, we need to run ”make install” to install the two new modules. Once the module is added, we need to enable the module.

Add “#!define WITH_PRESENCE” into the “kamailio.cfg” file. Then check if there is route logic defined for the “PRESENCE” module. By default there is a PRESENCE route defined in the default “kamailio.cfg” file. If it’s not there below is the route logic. Also we need to add “*route(PRESENCE)” in the ”request_route**” section.

# Presence server route
route[PRESENCE] {
        if(!is_method("PUBLISH|SUBSCRIBE"))
                return;

        if(is_method("SUBSCRIBE") && $hdr(Event)=="message-summary") {
                route(TOVOICEMAIL);
                # returns here if no voicemail server is configured
                sl_send_reply("404", "No voicemail service");
                exit;
        }

#!ifdef WITH_PRESENCE
        if (!t_newtran())
        {
                sl_reply_error();
                exit;
        }

        if(is_method("PUBLISH"))
        {
                handle_publish();
                t_release();
        } else if(is_method("SUBSCRIBE")) {
                handle_subscribe();
                t_release();
        }
        exit;
#!endif

        # if presence enabled, this part will not be executed
        if (is_method("PUBLISH") || $rU==$null)
        {
                xlog("@ 404 here 3");
                sl_send_reply("404", "Not here");
                exit;
        }
        return;
}

Now start the Kamailio server. Now we need to add some users. For that we can use ”kamctl” binary.

$ kamctl add user1@192.168.56.100 user1

$ kamctl add user2@192.168.56.100 user2

Let’s go ahead and test the settings. For testing we need some IM clients. In my testing, i’ve used Jitsi and adium IM clients. Once the accounts are configured on the clients, it’s better to start a packet capture using wireshark, so that we can see these PUBLISH, SUBSCRIBE and NOTIFY requests between the clients and the Kamailio server. If you are using Adium, go to account options and check the “Publish Status to Everyone” so that it will start sending PUBLISH request to the Kamailio server. Now add the accounts to the Contact lists on the IM clients and then we will be able to see the users status (ONline/OFFline). Now we can start the chat conversation between the user’s.