Docker, FreeSwitch, sipp, sippy_cup

Sippy_cup – FreeSwitch Load Test Simplified

Ever since the entry of Docker, everyone is busy porting their applications to Docker Containers. Now with the tools like Mesos, CoreOS etc we can easily achieve scalability also. @Plivo we always dedicate ourselves to play around such new technologies. In my previous blog posts, i’ve explained how to containerize the Freeswitch, how to perform some basic load test using simple dialplans etc. My previous load tests required a bunch of basic Freeswitch servers to originate calls to flood the calls to the FreeSwitch container. So this time i’m going to use a simple method, which everyone can use even from their laptops.

Enter SIPp. SIPp is a free Open Source test tool / traffic generator for the SIP protocol. But the main issue for beginer like me is in generating a proper XML for SIPp that can match to my exact production scenarios. After googling, i came across a super simple ruby wrapper over SIPp called sippy_cup. SIPpy_cup is a simple ruby wrapper over SIPp. We just need to create a simple yaml file and sippy_cup parses this yml file and generates the XML equivalent which will be then used to generate calls. sippy_cup can also be used to generate only the XML file for SIPp.

Setting up sippy_cup is very simple. There are only two dependencies

      1) ruby (2.1.2 recomended)
      2) SIPp

Another important dependency is our local internet bandwidth. Flooding too many calls will definitely result in network bottlenecks, which i faced when i generated 1k calls from my laptop. Now let’s install SIPp.

sudo apt-get install pcaputils libpcap-dev libncurses5-dev

wget 'http://sourceforge.net/projects/sipp/files/sipp/3.2/sipp.svn.tar.gz/download'

tar zxvf sipp.svn.tar.gz

# compile sipp
make

# compile sipp with pcapplay support
make pcapplay

Once we have installed SIPp and ruby, we can install sippy_cup via ruby gems.

gem install sippy_cup

Configuring sippy_cup

First we need to create yml file for our call flow. There is a good documentation available on the Readme on various options that can be used to create the yml to suit to our call flow. My call flow is pretty simple, i’ve a DialPlan in my Docker FS, which will play an mp3 file. So below is a simple yml config for this call flow

source: <local_machine_ip>
destination: <docker_fs_ip>:<fs_port>
max_concurrent: <no_of_concurrent_calls>
calls_per_second: <calls_per_second>
number_of_calls: <total_no_of_calls>
to_user: <to_number>            # => should match the FS Dialplan
steps:                      # call flow steps
  - invite                  # Initial Call INVITE
  - wait_for_answer         # Waiting for Answer, handles 100, 180/183 and finally 200 OK
  - ack_answer              # ACK for the 200 OK
  - sleep 1000              # Sleeps for 1000 seconds
  - send_bye                # Sends BYE signal to FS

Now let’s run sippy_cup using our config yml

sippy_cup -r test.yml

Below is the output of a sample load test. Total 20 calls with 10 concurrent calls

         INVITE ---------->      20        1         0
         100 <----------         20        0         0         0
         180 <----------         0         0         0         0
         183 <----------         0         0         0         0
         200 <----------  E-RTD1 20        0         0         0
         ACK ---------->         20        0
              [ NOP ]
         Pause [    30.0s]       20                            0
         BYE ---------->         20        0
------------------------------ Test Terminated --------------------------------


----------------------------- Statistics Screen ------- [1-9]: Change Screen --
  Start Time             | 2014-10-22   19:12:40.494470 1414030360.494470
  Last Reset Time        | 2014-10-22   19:13:45.355358 1414030425.355358
  Current Time           | 2014-10-22   19:13:45.355609 1414030425.355609
-------------------------+---------------------------+--------------------------
  Counter Name           | Periodic value            | Cumulative value
-------------------------+---------------------------+--------------------------
  Elapsed Time           | 00:00:00:000000           | 00:01:04:861000
  Call Rate              |    0.000 cps              |    0.308 cps
-------------------------+---------------------------+--------------------------
  Incoming call created  |        0                  |        0
  OutGoing call created  |        0                  |       20
  Total Call created     |                           |       20
  Current Call           |        0                  |
-------------------------+---------------------------+--------------------------
  Successful call        |        0                  |       20
  Failed call            |        0                  |        0
-------------------------+---------------------------+--------------------------
  Response Time 1        | 00:00:00:000000           | 00:00:01:252000
  Call Length            | 00:00:00:000000           | 00:00:31:255000
------------------------------ Test Terminated --------------------------------


I, [2014-10-22T19:13:45.357508 #17234]  INFO -- : Test completed successfully!

I tried to perform a large scale load test by making 1k calls with 250 concurrent calls. My local internet was flooding with network traffic as there was real Media packets coming from the servers, though it bottlenecked my internet, but still i was able to make 994 successfull calls. I suggest to do such heavy load test on machines wich has good network throughput. Below are the output for this test.

------------------------------ Scenario Screen -------- [1-9]: Change Screen --
  Call-rate(length)   Port   Total-time  Total-calls  Remote-host
   5.0(0 ms)/1.000s   8836     585.61 s         1000  54.235.170.44:5060(UDP)

  Call limit reached (-m 1000), 0.507 s period  1 ms scheduler resolution
  6 calls (limit 250)                    Peak was 176 calls, after 150 s
  0 Running, 8 Paused, 1 Woken up
  604 dead call msg (discarded)          0 out-of-call msg (discarded)
  3 open sockets
  1490603 Total RTP pckts sent           0.000 last period RTP rate (kB/s)

                                 Messages  Retrans   Timeout   Unexpected-Msg
         INVITE ---------->      1000      332       0
         100 <----------         954       53        0         0
         180 <----------         0         0         0         0
         183 <----------         0         0         0         0
         200 <------2014-10-22  19:19:23.202714 1414030763.202714: Dead call 990-17510@192.168.1.146 (successful), 

received 'SIP/2.0 200 OK
Via: SIP/2.0/UDP 192.168.1.146:8836;received=208.66.27.62;branch=z9hG4bK-17510-990-8
From: "sipp" <sip:sipp@192.168.1.146>;tag=990
To: <sip:14158872327@54.235.170.44:5060>;tag=9p6t351mvXZXg
Call-ID: 990-17510@192.168.1.146
CSeq: 2 BYE
User-Agent: Plivo
Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REFER, NOTIFY
Supported: timer, precondition, path, replaces
Conte----  E-RTD1        994       126       0         0
         ACK ---------->         994       126
              [ NOP ]
       Pause [    30.0s]         994                           0
         BYE ---------->         994       0
------------------------------ Test Terminated --------------------------------


----------------------------- Statistics Screen ------- [1-9]: Change Screen --
  Start Time             | 2014-10-22   19:15:29.941276 1414030529.941276
  Last Reset Time        | 2014-10-22   19:25:15.056475 1414031115.056475
  Current Time           | 2014-10-22   19:25:15.564038 1414031115.564038
-------------------------+---------------------------+--------------------------
  Counter Name           | Periodic value            | Cumulative value
-------------------------+---------------------------+--------------------------
  Elapsed Time           | 00:00:00:507000           | 00:09:45:622000
  Call Rate              |    0.000 cps              |    1.708 cps
-------------------------+---------------------------+--------------------------
  Incoming call created  |        0                  |        0
  OutGoing call created  |        0                  |     1000
  Total Call created     |                           |     1000
  Current Call           |        6                  |
-------------------------+---------------------------+--------------------------
  Successful call        |        0                  |      994
  Failed call            |        0                  |        0
-------------------------+---------------------------+--------------------------
  Response Time 1        | 00:00:00:000000           | 00:00:01:670000
  Call Length            | 00:00:00:000000           | 00:00:31:673000
------------------------------ Test Terminated --------------------------------

Sippy_cup is definitely a good tool for all beginers who finds really hard time to work around with SIPp XML’s. I’m really excited to see how Docker is going to contirbute to VOIP world.

CollectD, Elasticsearch, Kibana, logstash, Monitoring, Redis

Monitoring Redis Using CollectD and ELK

Redis is an open-source, networked, in-memory, key-value data store. It’s being heavily used every where from Web stack to Monitoring to Message queues. Monitoring tools like Sensu already has some good scripts to Monitor Redis. Last Month during PyCon 2014 @Plivo, opensourced a new rate limited queue called SHARQ which is based on Redis. So apart from just Monitoring checks, we decided to have a tsdb of what’s happening in our Redis Cluster. Since we are heavily using ELK stack to visualize our infrastructure, we decided to go ahead with the same.

CollectD Redis Plugin

There is a cool CollectD plugin for Redis. It pulls a verity of Data from Redis which includes, Memory used, Commands Processed, No. of Connected Clients and slaves, No. of blocked Clients, No. of Keys stored/db, uptime and challenges since last save. The installation is pretty simple and straight forward.

$ apt-get update && apt-get install collectd

$ git clone https://github.com/powdahound/redis-collectd-plugin.git /tmp/redis-collectd-plugin

Now place the redis_info.py file onto the collectd folder and enable the Python Plugins so that collectd can use this python file. Below is our collectd conf

Hostname    "<redis-server-fqdn>"
Interval 10
Timeout 4
Include "/etc/collectd/filters.conf"
Include "/etc/collectd/thresholds.conf"
LoadPlugin network
ReportStats true

        LogLevel info

Include "/etc/collectd/redis.conf"      # This is the configuration for the Redis plugin
<Plugin network>
    Server "<logstash-fqdn>" "<logstash-collectd-port>"
</Plugin>

Now copy the redis python plugin and the conf file to collectd folder.

$ mkdir /etc/collectd/plugin            # This is where we are going to place our custom plugins

$ cp /tmp/redis-collectd-plugin/redis_info.py /etc/collectd/plugin/

$ cp /tmp/redis-collectd-plugin/redis.conf /etc/collectd/

By default, the plugin folder in the redis.conf is defined as ‘/opt/collectd/lib/collectd/plugins/python’. Make sure to replace this with the location where we are copying the plugin file, in our case “/etc/collectd/plugin”. Now lets restart the collectd daemon to enable the redis plugin.

$ /etc/init.d/collectd stop

$ /etc/init.d/collectd start

In my previous Blog, i’ve mentioned how to enable and use the ColectD input plugin in Logstash and to use Kibana to plot the data coming from the collectd. Below are the Data’s that we are receiving from the CollectD on Logstash,

  1) type_instance: blocked_clients
  2) type_instance: evicted_keys
  3) type_instance: connected_slaves
  4) type_instance: commands_processed
  5) type_instance: connected_clients
  6) type_instance: used_memory 
  7) type_instance: <dbname>-keys
  8) type_instance: changes_since_last_save
  9) type_instance: uptime_in_seconds
10) type_instance: connections_received

Now we need to Visualize these via Kibana. Lets create some ElasticSearch queries so that visualize them directly. Below are some sample queries created in Kibana UI.

1) type_instance: "commands_processed" AND host: "<redis-host-fqdn>"
2) type_instance: "used_memory" AND host: "<redis-host-fqdn>"
3) type_instance: "connections_received" AND host: "<redis-host-fqdn>"
4) type_instance: "<dbname>-keys" AND host: "<redis-host-fqdn>"