Ansible

Ping Your Ansible From Slack

We have been using Ansible for all our Deployments, Code Updates, Inventory Management. We have also built a higher level API for Ansible called bootstrapper. Bootstrapper provides a rest API for executing various complex tasks via Ansible. Also, there is a simple python CLI for bootstrapper called bootstrappercli which interacts with the bootstrapper servers in our prod/staging env and performs the tasks. We have been using slack for our team interactions and it became a part of my chrome tabs always. We also have a custom slack callback plugin, that provides us a real time callback for each step of playbook execution.

Since bootstrapper is providing a sweet API, we decided to make Slack to directly talk to Ansible,. After googling for some time, i came across a simple slackbot called limbo, written in Python(offcourse) and the best part is its plugin system. It’s super easy to write custom plugins. As an initial step, i wrote a couple of plugins for performing codepush, adhoc and EC2 Management. Below are some sample plugin codes,

Code-Update Plugin
"""Only direct messages are accepted, format is @user codeupdate <tag> <env>"""

import requests
import json
import unicodedata

ALLOWED_CHANNELS = []
ALLOWED_USERS = ['xxxxx']
ALLOWED_ENV = ['xxx', 'xxx']
ALLOWED_TAG = ['xxx', 'xxx']

BOOTSTRAPPER__PROD_URL = "http://xxx.xxx.xxx.xxx:yyyy"
BOOTSTRAPPER_STAGING_URL = "http://xxx.xxx.xxx.xxx:yyyy"

def get_url(infra_env):
    if infra_env == "prod":
        return BOOTSTRAPPER__PROD_URL
    else:
        return BOOTSTRAPPER_STAGING_URL

def codeupdate(ans_env, ans_tag):
    base_url = get_url(ans_env)
    req_url = base_url + '/ansible/update-code/'
    resp = requests.post(req_url, data={'tags': ans_tag, 'env': ans_env})
    return [resp.status_code, resp.text]

def on_message(msg, server):
    init_msg =  unicodedata.normalize('NFKD', msg['text']).encode('ascii','ignore').split(' ')[0]
    cmd = unicodedata.normalize('NFKD', msg['text']).encode('ascii','ignore').split(' ')[1]
    if cmd != 'codeupdate':
        return ""
    ### slight hackish method to check if the message is a direct message or not
    elif init_msg == '<@xxxxxx>:':    # Message is actually a direct message to the Bot
        orig_msg =  unicodedata.normalize('NFKD', msg['text']).encode('ascii','ignore').split(' ')[2:]
        msg_user = msg['user']
        msg_channel = msg['channel']
        if msg_user in ALLOWED_USERS and msg_channel in ALLOWED_CHANNELS:
            env = orig_msg[0]
            if env not in ALLOWED_ENV:
                return "Please Pass a Valid Env (xxxx/yyyy)"
            tag = orig_msg[1]
            if tag not in ALLOWED_TAG:
                return "Please Pass a Valid Tag (xxxx/yyyy)"
            return codeupdate(env, tag)
        else:
            return ("!!! You are not Authorized !!!")
    else:
        return "invalid Message Format, Format: <user> codeupdate <env> <tag>"
Adhoc Execution plugin
"""Only direct messages are accepted, format is <user> <env> <host> <mod> <args_if_any"""

import requests
import json
from random import shuffle
import unicodedata

    ALLOWED_CHANNELS = []
    ALLOWED_USERS = ['xxxxx']
    ALLOWED_ENV = ['xxx', 'xxx']
    ALLOWED_TAG = ['xxx', 'xxx']

    BOOTSTRAPPER__PROD_URL = "http://xxx.xxx.xxx.xxx:yyyy"
    BOOTSTRAPPER_STAGING_URL = "http://xxx.xxx.xxx.xxx:yyyy"

def get_url(infra_env):
    if infra_env == "prod":
        return BOOTSTRAPPER__PROD_URL
    else:
        return BOOTSTRAPPER_STAGING_URL

def adhoc(ans_host, ans_mod, ans_arg, ans_env):
    base_url = get_url(ans_env)
    req_url = base_url + '/ansible/adhoc/'
    if ans_arg is None:
        resp = requests.get(req_url, params={'host': ans_host, 'mod': ans_mod})
    else:
        resp = requests.get(req_url, params={'host': ans_host, 'mod': ans_mod, 'args': ans_arg})
    return resp.text

def on_message(msg, server):
### slight hackish method to check if the message is a direct message or not
    init_msg =  unicodedata.normalize('NFKD', msg['text']).encode('ascii','ignore').split(' ')[0]
    cmd = unicodedata.normalize('NFKD', msg['text']).encode('ascii','ignore').split(' ')[1]
    if cmd != 'adhoc':
        return ""
    elif init_msg == '<@xxxxxxx>:':    # Message is actually a direct message to the Bot
        orig_msg =  unicodedata.normalize('NFKD', msg['text']).encode('ascii','ignore').split(' ')[2:]
        msg_user = msg['user']
        msg_channel = msg['channel']
        if msg_user in ALLOWED_USERS and msg_channel in ALLOWED_CHANNELS:
            env = orig_msg[0]
            if env not in ALLOWED_ENV:
                return "Only Dev Environment is Supported Currently"
            host = orig_msg[1]
            mod = orig_msg[2]
            arg = orig_msg[3:]
            return adhoc(host, mod, arg, env)
        else:
            return ("!!! You are not Authorized !!!")
    else:
        return "Invalid Message, Format: <user> <env> <host> <mod> <args_if_any>"

Configure Limbo as per the Readme present in the Repo and start the limbo binary. Now let’s try executing some Ansible operations from Slack.

Let’s try a simple ping module

Let’s try updating a staging cluster,

WooHooo, it worked ;) This is opening up a new dawn for us in managing our infrastructure from Slack. But offcourse ACL is an important factor, as we should restrict access to specific operations. Currently, this ACL logic is written with in the plugin itself. It checks the user who is executing the command and from which channel he is executing, if both matches only, the Bot will start the talking to Bootstrapper, or else it throws an error to the user. I’m pretty much excited to play with this, looking forward to automate Ansible tasks as much as possible from Slack :)

Standard
etcd, serf

Building Self Discovering Clusters With ETCD and SERF

Now a days its very difficult to contain all the services in a single box. Also, companies have started adopting micro service based architectures. For Docker user’s, they can achieve a highly scalable system like Amazon ASG using Apache mesos and their supporting frameworks like Marathon etc… But there are a lot services, where in a new node needs to know the existing members in a cluster to join successfully. Common method will be having a static ip’s to the machines. But we have started moving all our clusters slowly to become immutable via ASG. So either we need to go for Elastic IP’s, and use these static ip’s, but the next challenge would, what if we want to scale the cluster up. Also, if these instances are inside a private network, assigning static local ip’s is also difficult. So if a new machine is launched or if asg relaunches a faulty machine, these machine’s should know the existing cluster nodes. But again not all applications provide service-discovery features. I was testing around with a new Erlang based MQTT broker called verne. For a new Verne node to join the cluster, we need to pass the ip’s of all existing nodes to the JOIN command.

So a quick solution came to mind was using a Distributed K-V pair system like etcd. Idea was to use etcd to have a key-value that can tell us the nodes present in a cluster. But quickly i found another hurdle. Imagine a node belonging to a cluster, went down, the key will be still present in etcd. If we have something like ASG, will launch a new machine, so this new machine’s IP will not be present in the etcd key. We can delete manaully and launch a new machine, then add its IP to the etcd. But our ultimate goal is to automate all these, otherwise our clusters cannot become a full immutable and auto healing.

So we need a system, that can keep track of our Node’s activity atleast, when they join/leave the cluster.

SERF

Enter SERF. Serf is a decentralized solution for cluster membership, failure detection, and orchestration. Serf relies on an efficient and lightweight gossip protocol to communicate with nodes. The Serf agents periodically exchange messages with each other in much the same way that a zombie apocalypse would occur: it starts with one zombie but soon infects everyone. Serf is able to quickly detect failed members and notify the rest of the cluster. This failure detection is built into the heart of the gossip protocol used by Serf.

Serf, also need to know IP’s of other Serf Agents in the cluster. So initially when we start the cluster, we know the IP’s of the other Serf agents, and the serf agents will then populate our Cluster Key’s, on ETCD. Now if a new machine is launched/relaunched, etcd will use the Discovery URL and will connect to the ETCD Cluster. Once it has joined the existing cluter, our Serf agent will start and it will talk to the etcd agent running locally and will create the necessary key for the node. Once the Key is being created, Serf Agent will join the serf cluster using one of the existing nodes IP that was present in the ETCD cluster.

ETCD Discovery URL

A bit about etcd Discovery URL. It’s an inbuild discovery method in etcd for identifying the dynamic members in an etcd cluster. First, we need an ETCD server, this server won’t be a part of our cluster, instead this is used only for providing the discovery URL. We can use this server for multiple ETCD clusters, provided each cluster should have a unique URL. So all the new nodes can talk to their unique url and can join with the rest of the etcd nodes.

# For starting the ETCD Discovery Server

$ ./etcd -name test --listen-peer-urls "http://172.16.16.99:2380,http://172.16.16.99:7001" --listen-client-urls "http://172.16.16.99:2379,http://172.16.16.99:4001" --advertise-client-urls "http://172.16.16.99:2379,http://172.16.16.99:4001" --initial-advertise-peer-urls "http://172.16.16.99:2380,http://172.16.16.99:7001" --initial-cluster 'test=http://172.16.16.99:2380,test=http://172.16.16.99:7001'

Now we need to create a unique Discovery URL for our clusters.

$ curl -X PUT 172.16.16.99:4001/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83/_config/size -d value=3

 where, 6c007a14875d53d9bf0ef5a6fc0257c817f0fb83 => a unique UUID
        value => Size of the cluster, if the number of etcd nodes that tries to join with the specific url is > value, then the extra etcd processes will fall back to being proxies by default.

SERF Events

Serf comes with inbuilt event handler system. It has some inbuilt event handlers for Member Events like member-join, member-leave. We are going to concentrate on these two events. Below is a simple handler script, that will add/remove keys in ETCD, whenever a node joins/leave the cluster.

# handler.py
#!/usr/bin/python
import sys
import os
import requests
import json

base_url = "http://<ip-of-the-node>:4001"   # Use any config mgmt tool or user-data script to populate this value for each host
base_key = "/v2/keys"
cluster_key = "/cluster/<cluster-name>/"    # Use any config mgmt tool or user-data script to populate this value for each host

for line in sys.stdin:
    event_type = os.environ['SERF_EVENT']
    if event_type == 'member-join':
        address = line.split('\t')[1]
        full_url = base_url + base_key + cluster_key + address
        r = requests.get(full_url)
        if r.status_code == 404:
            r = requests.put(full_url, data="member")     # If the key doesn't exists, it will create a new key
            if r.status_code == 201:
                print "Key Successfully created in ETCD"
            else:
                print "Failed to create the Key in ETCD"
        else:
            print "Key Already exists in ETCD"
    if event_type == 'member-leave':
        address = line.split('\t')[1]
        full_url = base_url + base_key + cluster_key + address
        r = requests.get(full_url)
        if r.status_code == 200:                         # If the node leaves the cluster and the key still exists, remove it
            r = requests.delete(full_url)
        if r.status_code == 404:            
                print "Key already removed by another Serf Agent"
            else:
                print "Key Successfully removed from ETCD"
        else:
            print "Key already removed from ETCD"

Design

Now we have some vague idea of serf and etcd. Let’s start building our self-discovering cluster. Make sure that the etcd Discovery node is up and a unique URL has been generated for the same. Once the URL is ready, lets start our ETCD cluster.

# Node1

$ /etcd -name test1 --listen-peer-urls "http://172.16.16.102:2380,http://172.16.16.102:7001" --listen-client-urls "http://172.16.16.102:2379,http://172.16.16.102:4001" --advertise-client-urls "http://172.16.16.102:2379,http://172.16.16.102:4001" --initial-advertise-peer-urls "http://172.16.16.102:2380,http://172.16.16.102:7001" --discovery http://172.16.16.99:4001/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83/

# Sample Output

2015/06/01 19:51:10 etcd: listening for peers on http://172.16.16.102:2380
2015/06/01 19:51:10 etcd: listening for peers on http://172.16.16.102:7001
2015/06/01 19:51:10 etcd: listening for client requests on http://172.16.16.102:2379
2015/06/01 19:51:10 etcd: listening for client requests on http://172.16.16.102:4001
2015/06/01 19:51:10 etcdserver: datadir is valid for the 2.0.1 format
2015/06/01 19:51:10 discovery: found self a9e216fb7b8f3283 in the cluster
2015/06/01 19:51:10 discovery: found 1 peer(s), waiting for 2 more      # Since our cluster Size is 3, and this is the first node, its waiting for other 2 nodes to join

Lets start the other two nodes,

./etcd -name test2 --listen-peer-urls "http://172.16.16.101:2380,http://172.16.16.101:7001" --listen-client-urls "http://172.16.16.101:2379,http://172.16.16.101:4001" --advertise-client-urls "http://172.16.16.101:2379,http://172.16.16.101:4001" --initial-advertise-peer-urls "http://172.16.16.101:2380,http://172.16.16.101:7001" --discovery http://172.16.16.99:4001/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83/

./etcd -name test3 --listen-peer-urls "http://172.16.16.100:2380,http://172.16.16.100:7001" --listen-client-urls "http://172.16.16.100:2379,http://172.16.16.100:4001" --advertise-client-urls "http://172.16.16.100:2379,http://172.16.16.100:4001" --initial-advertise-peer-urls "http://172.16.16.100:2380,http://172.16.16.100:7001" --discovery http://172.16.16.99:4001/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83/

Now we have the ETCD cluster up and running, lets configure the serf.

$ ./serf agent -node=test1 -bind=172.16.16.102:5000 -rpc-addr=172.16.16.102:7373 -log-level=debug -event-handler=/root/handler.py

$ ./serf agent -node=test2 -bind=172.16.16.101:5000 -rpc-addr=172.16.16.101:7373 -log-level=debug -event-handler=/root/handler.py

$ ./serf agent -node=test3 -bind=172.16.16.100:5000 -rpc-addr=172.16.16.100:7373 -log-level=debug -event-handler=/root/handler.py

Now these 3 serf agents are running in standalone mode and they are not aware of each other. Lets make the agents to join and form the serf cluster.

$ ./serf join -rpc-addr=172.16.16.100:7373 172.16.16.101:5000

$ ./serf join -rpc-addr=172.16.16.102:7373 172.16.16.100:5000

Let’s query the serf agents and see if the cluster has been created.

$ root@sentinel:~# ./serf members -rpc-addr=172.16.16.100:7373
  test3  172.16.16.100:5000  alive
  test2  172.16.16.101:5000  alive
  test1  172.16.16.102:5000  alive

    $ root@sentinel:~# ./serf members -rpc-addr=172.16.16.101:7373
      test3  172.16.16.100:5000  alive
      test2  172.16.16.101:5000  alive
      test1  172.16.16.102:5000  alive

    $ root@sentinel:~# ./serf members -rpc-addr=172.16.16.102:7373
      test3  172.16.16.100:5000  alive
      test2  172.16.16.101:5000  alive
      test1  172.16.16.102:5000  alive

Now our serf nodes are up, let’s check our etcd cluster to see if serf has created the keys.

$ ./etcdctl -C 172.16.16.100:4001 ls /cluster/test/
  /cluster/test/172.16.16.100
  /cluster/test/172.16.16.101
  /cluster/test/172.16.16.102

Serf has successfully created the key. Now, let’s terminate one of the node’s. one of our Serf agent will immidiately receives an EVENT for member-leave and it will start the handler. Below is the log for the First agent who received the event.

2015/06/01 22:56:43 [INFO] serf: EventMemberLeave: test1 172.16.16.102
2015/06/01 22:56:44 [INFO] agent: Received event: member-leave
2015/06/01 22:56:45 [DEBUG] agent: Event 'member-leave' script output: member-leave => 172.16.16.102
Key Successfully removed from ETCD                                
2015/06/01 22:56:45 [DEBUG] memberlist: Initiating push/pull sync with: 172.16.16.100:5000

This event will later gets passed to the other agent,

2015/06/01 22:56:43 [INFO] serf: EventMemberLeave: test1 172.16.16.102
2015/06/01 22:56:44 [INFO] agent: Received event: member-leave
2015/06/01 22:56:45 [DEBUG] agent: Event 'member-leave' script output: member-leave => 172.16.16.102
Key already removed by another Serf Agent

Let’s make sure that the KEY was successfully removed from ETCD.

$ ./etcdctl -C 172.16.16.100:4001 ls /cluster/test/
  /cluster/test/172.16.16.100
  /cluster/test/172.16.16.101

Now let’s bring the old node back online,

$ ./serf agent -node=test1 -bind=172.16.16.102:5000 -rpc-addr=172.16.16.102:7373 -log-level=debug -event-handler=/root/handler.py
  ==> Starting Serf agent...
  ==> Starting Serf agent RPC...
  ==> Serf agent running!
           Node name: 'test1'
           Bind addr: '172.16.16.102:5000'
            RPC addr: '172.16.16.102:7373'
           Encrypted: false
            Snapshot: false
             Profile: lan

  ==> Log data will now stream in as it occurs:

      2015/06/01 23:01:55 [INFO] agent: Serf agent starting
      2015/06/01 23:01:55 [INFO] serf: EventMemberJoin: test1 172.16.16.102
      2015/06/01 23:01:56 [INFO] agent: Received event: member-join
      2015/06/01 23:01:56 [DEBUG] agent: Event 'member-join' script output: member-join => 172.16.16.102
  Key Successfully created in ETCD

The standalone agent has already created the key. Now let’s make the Serf agent to join the cluster. It can use any one of the IP from the K-V apart from it’s own IP

$ ./serf join -rpc-addr=172.16.16.102:7373 172.16.16.100:5000
  Successfully joined cluster by contacting 1 nodes.

Once the node joins the cluster, the other existing serf nodes will receive the member-join event, and will also tries to execute the handler. But since the key is already being created, it wont recereate it.

    2015/06/01 23:04:56 [INFO] agent: Received event: member-join
    2015/06/01 23:04:57 [DEBUG] agent: Event 'member-join' script output: member-join => 172.16.16.102
Key Already exists in ETCD

Let’s check our ETCD cluster to see if the keys are recreated.

$ ./etcdctl -C 172.16.16.100:4001 ls /cluster/test/
  /cluster/test/172.16.16.100
  /cluster/test/172.16.16.101
  /cluster/test/172.16.16.102

So now we have IP’s of our active nodes in the cluster in etcd. Now we need to extract the IP’S from etcd and need to use them with our Applications. Below is a simple python script that returns the IP’s in as a python LIST.

import requests
import json

base_url = "http://<local_ip_of_server>:4001"
base_key = "/v2/keys"
cluster_key = "/cluster/<cluster_name>/"
full_url = base_url + base_key + cluster_key
r = requests.get(full_url)
json_data = json.loads(r.text)
node_list = json_data['node']['nodes']
nodes = []
for i in node_list:
   nodes.append(i['key'].split(cluster_key)[1])
print nodes

We can use ETCD and confd to generate the config’s dynamically for our applications.

Conclusion

ETCD and SERF provide a powerfull combination for achieving a self-discovery feature onto our clusters. But it still needs heavy QA testing before it’s being used in production. Especially, we need to test on the confd automation part more, as the we are not using a single KEY-VALUE, as the SERF creates a separate K-V for each node. And the IP discovery from ETCD is done by iterating the key-directory. But these tools are backed by an awesome community, so i’m sure soon these tools will helps us to achieve a whole new level for clustering our applications.

Standard
Ansible

Building Auto Healing Clusters With AWS and Ansible

As we all know this is the era of cloud servers. With the emergence of cloud, no need to worry about the difficulties in hosting servers on premises. But if you are cloud engineer, you definitely know that any thing can happen to your machine. Unlike going and fixing on our own, in cloud its difficult. Even i faced a lot of such weird issues, where my cloud service provider terminated my server’s which includes my Postgres DB master also. So having a self healing cluster will help us a lot, especially if the server goes down in the middle of our sleep. Stateless services are the easiest candidates for self healing compared to DB’s, especially if we are using a Master-Slave DB architecture.

For those who are using Docker and Mesos, Marathon provides similar scaling features like Amazon ASG. We define the number of instances that has to be running, and Marathon makes sure that number always exists. Like Amazon ASG, it will relaunch a new container, if any container accidentally terminates. I’ve personally tested this feature of Marathon long back, and it’s really a promising one. There are indeed other automated container management systems, but the marathon is quite flexible to me.

But in Clementine, in our current architecture, we are not yet using Docker in Production and we heavily use AWS for all our clusters. With more features like Secure messaging, VOIP etc.. added to our product, we are expanding tremendously. And so does our infrastructure. Being a DevOps engineer, i need to keep the uptime. So this time i decided to prototype a self healing cluster using Amazon ASG and Ansible.

Design

For the auto healing i’m going to use Amazon ASG and Ansible. Since Ansible is a client less application, we need to either use Ansible in stand-alone mode and provision the machine via cloud init script, or use the ansible-pull. Or as the company recommends, use Ansible Tower, which is a paid solution. But we have built our own higher level API solution over Ansible called bootstrapper. Bootstrapper exposes a higher level rest API which we can invoke for all our Ansible management. Our in house version of Bootstrapper can perform various actions like, ec2 instance launch with/without EIP, Ahdoc command execution, server bootstrapping, code update etc ….

But again, if we use a plain AMI and tries to bootstrap the server completely during startup, it puts a heavy delay, especially when pypi gives u time out while installing the pip packages. So we decided to use a custom AMI which has our latest build in it. Jenkins takes care of this part. Our build flow is like this,

Dev pushes code to Master/Dev => Jenkins performs build test => if build succeeds, starts our master/dev packages => uploads the package to our APT repo => Packer builds the latest image via packer-aws-chroot

While building the image, we add two custom scripts on to our images, 1) setup_eip.sh (manages EIP for the instance via Bootstrapper), 2) ans_bootstrap.sh (Manages server bootstrapping via Ansible)

# set_eip.sh
inst_id=`curl -s http://169.254.169.254/latest/meta-data/instance-id`
role=`cat /etc/ansible/facts.d/clem.fact  | grep role | cut -d '=' -f2`  # our custom ansible facts
env=`cat /etc/ansible/facts.d/clem.fact  | grep env | cut -d '=' -f2`    # our custom ansible facts
if [ $env == "staging" ]; then
  bootstrapper_url="xxx.xxx.xxx.xxx:yyyy/ansible/set_eip/"
else
  bootstrapper_url="xxx.xxx.xxx.xxx:yyyy/ansible/set_eip/"     # our bootstrapper api for EIP management
fi
curl -X POST -s -d "instance_id=$inst_id&role=$role&env=$env" $bootstrapper_url

The above POST request to Ansible performs EIP management and Ansible will assign the proper EIP to the machine without any collision. We keep an EIP mapping for our cluster, which makes sure that we are not assigning any wrong EIP to the machines. If no EIP is available, we raise an exception and email/slack the infra team about the instance and cluster

# ans_bootstrap.sh
local_ip=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`
    role=`cat /etc/ansible/facts.d/clem.fact  | grep role | cut -d '=' -f2`  # our custom ansible facts
    env=`cat /etc/ansible/facts.d/clem.fact  | grep env | cut -d '=' -f2`    # our custom ansible facts
    if [ $env == "staging" ]; then
      bootstrapper_url="xxx.xxx.xxx.xxx:yyyy/ansible/role/"
    else
      bootstrapper_url="xxx.xxx.xxx.xxx:yyyy/ansible/role/"     # our bootstrapper api for role based playbook execution, multiple roles can be passed to the API
    fi
curl -X POST -d "host=$local_ip&role=$role&env=$env" $bootstrapper_url

These two scripts are executed via cloud-init script during machine bootup. Once we have the image ready, we need to create a launch config for the ASG. Below is a sample userdata script,

#! /bin/bash

echo "Starting EIP management via Bootstrapper"
/usr/local/src/boostrap_scripts/set_eip.sh
echo "starting server bootstrap"
/usr/local/src/bootstrap_scripts/ans_bootstrap.sh

Now create an Autoscaling group with the required number of nodes. On the scaling policies, select Keep this group at its initial size. Once the ASG is up, it will start the nodes based on the AMI and Subnet mentioned. Once the machine starts booting, cloud-init script will start executing our userdata scripts, which in turn talks to our Bootstrapper-Ansible and starts assigning EIP and executing the playbooks onto the hosts. Below is a sample log on our bootstrapper for EIP management, invoked by an ASG node while it was booting up.

01:33:11 default: bootstrap.ansble_set_eip(u'staging', u'<ansible_role>', u'i-xxxxx', '<remote_user>', '<remote_key>') (4df8aee9-ab0e-4152-973a-b227ddac91a1)
EIP xxx.xxx.xxx.xxx is attached to instance i-xyxyxyxy
EIP xxx.xxx.xxx.xxx is attached to instance i-xyxyxyxy
EIP xxx.xxx.xxx.xxx is attached to instance i-xyxyxyxy
EIP xxx.xxx.xxx.xxx is attached to instance i-xyxyxyxy
EIP xxx.xxx.xxx.xxx is attached to instance i-xyxyxyxy
Free EIP available for <our-cluster-name> is xxx.xxx.xxx.xxx   # Bootstrapper found the free ip that is allowed to be assigned for this particular node

PLAY [localhost] **************************************************************

TASK: [adding EIP to the instance] ********************************************
changed: [127.0.0.1]
01:33:12 Job OK, result = {'127.0.0.1': {'unreachable': 0, 'skipped': 0, 'ok': 2, 'changed': 1, 'failures': 0}}

I’ve tested this prototype with one of our VOIP clusters and the cluster is working is perfectly with the corresponding EIP’s as mapped. We terminated the machines, multiple times, to make sure that the EIP management is working properly and the servers are getting bootstrapped. The results are promising and this now motivates us to migrate all of our stateless clusters onto self healing so that our cluster auto heals whenever a machine becomes unhealthy. No need of any Human intervention unless Amazon really screws their ASG :p

Standard
Ansible, Redis

Building an Automated Config Management Server using Ansible+Flask+Redis

It’s almost 2 months since i’ve started playing full time on ansible. Like most of the SYS-Admin’s, ive been using ansible via cli most of the time. Unlike Salt/Puppet, ansible is an agent less one. So we need to invoke things from the box which contains ansible and the respective playbooks installed. Also, if you want to use ansible with ec2 features like auto-scaling, we need to either buy Ansible Tower, or need to use ansible-fetch along with the userdata script. I’ve also seen people, who uses custom scripts, that fetches their repo and execute ansible playbook locally to bootstrap.

Being a good fan of Flask, i’ve used flask on creating many backend API’s to automate a bunch of my tasks. So this time i decided to write a simple Flask API for executing Ansible playbook/ Ansible Adhoc commands etc.. Ansible also provides a Python API, which also made my work easier. Like most of the Ansible user’s, i use Role’s for all my playbooks. We can directly expose an API to ansible and can execute playbooks. But there are cases, where the playbook execution takes > 5min, and offcourse if there is any network latency it will affect our package download etc. I don’t want to force my HTTP clients to wait for the final output of the playbook execution to get a response back.

So i decided to go ahead with a JOB Queue feature. Each time a request comes to my API, the job is queued in Redis and the JOB ID will be returned to the clients. Then my job-workers pick the job’s from the redis queue and performs the job execution on the backend and workers will keep updating the job status. So now, i need to expose 2 API’s first, ie, one for receiving jobs and one for job status. For Redis Queue, there is an awesome library called rq. I’ve been using rq for all queuing tasks.

Flask API

The JOB accepts a bunch of parameters like host, role, env via HTTP POST method. Since the role/host etc.. have to be retrieved from the HTTP request, my playbook yml file has to be a dynamic one. So i’ve decided to use Jinja templating to dynamically create my playbook yml file. Below is my sample API for Role based playbook execution.

@app.route('/ansible/role/', methods=['POST'])
def role():
  inst_ip = request.form['host']                          # Host to which the playbook has to be executed
  inst_role = request.form['role']                        # Role to be applied on the Playbook
  env = request.form['env']               # Extra evn variables to be passed while executing the playbook
  ans_remote_user = "ubuntu"                  # Default remote user
  ans_private_key = "/home/ubuntu/.ssh/id_rsa"        # Default ssh private key
  job = q.enqueue_call(                   # Queuing the job on to Redis
            func=ansble_run, args=(inst_ip, inst_role, env, ans_remote_user, ans_private_key,), result_ttl=5000, timeout=2000
        )
  return job.get_id()                     # Returns job id if the job is successfully queued to Redis

Below is a sample templating function that generates the playbook yml file via Jinja2 templating

def gen_pbook_yml(ip, role):
  r_text = ''
  templateLoader = jinja2.FileSystemLoader( searchpath="/" )
  templateEnv = jinja2.Environment( loader=templateLoader )
  TEMPLATE_FILE = "/opt/ansible/playbook.jinja"                # Jinja template file location
  template = templateEnv.get_template( TEMPLATE_FILE )
  role = role.split(',')                       # Make Role as an array if Multiple Roles are mentioned in the POST request
  r_text = ''.join([random.choice(string.ascii_letters + string.digits) for n in xrange(32)])  
  temp_file = "/tmp/" + "ans-" + r_text + ".yml"           # Crating a unique playbook yml file
  templateVars = { "hst": ip,
                   "roles": role
                 }
  outputText = template.render( templateVars )             # Rendering template
  text_file = open(temp_file, "w")
  text_file.write(outputText)                      # Saving the template output to the temp file
  text_file.close()
  return temp_file

Once the playbook file is ready, we need to invoke Ansible’s API to perform our bootstrapping. This is actually done by the Job workers. Below is a sample function which invokes the playbook API from Ansible CORE.

def ansble_run(ans_inst_ip, ans_inst_role, ans_env, ans_user, ans_key_file):
  yml_pbook = gen_pbook_yml(ans_inst_ip, ans_inst_role)   # Generating the playbook yml file
  run_pbook = ansible.playbook.PlayBook(          # Invoking Ansible's playbook API
                 playbook=yml_pbook,
                 callbacks=playbook_cb,
                 runner_callbacks=runner_cb,
                 stats=stats,
                 remote_user=ans_user,
                 private_key_file=ans_key_file,
                 host_list="/etc/ansible/hosts",          # use either host_file or inventory
#                Inventory='path/to/inventory/file',
                 extra_vars={
                    'env': ans_env
                 }
                 ).run()
  return run_pbook                    # We can tune the output that has to be returned

Now the job-workers executes and updates the status on the Redis. Now we need to expose our JOB status API. Below is a sample Flask API for the same.

@app.route("/ansible/results/<job_key>", methods=['GET'])
def get_results(job_key):

    job = Job.fetch(job_key, connection=conn)
    if job.is_finished:
        ret = job.return_value
    elif job.is_queued:
        ret = {'status':'in-queue'}
    elif job.is_started:
        ret = {'status':'waiting'}
    elif job.is_failed:
        ret = {'status': 'failed'}

    return json.dumps(ret), 200

Now, we have a fully fledged API server for executing Role based playbooks. This API can also be used with user data scripts in autoscaling, where in we need to perform an HTTP POST request to the API server, and our API server will start the Bootstrapping. I’ve tested this app locally with various scenarios and the results are promising. Now as a next step, i’m planning to extend the API to do more jobs like, automating Code Pushes, Running AD-Hoc commands via API etc… With applications like Ansible, Redis, Flask, i’m sure SYS Admins can attain the DevOps Nirvana :). I’ll be pushing the latest working code to my Github account soon…

Standard
Docker, Jenkins

Virtual Cluster Testing Using Jenkins and Docker

Nowadays CI or Conitnous Integration is being implemented in almost all IT companies. Many of the DevOps work’s are in related to the CI. The common scenario is, Developers push the codes to the GIT/SVN repo and triggers jenkins to perform tests and sometimes packaging, and if it’s a fuly automated system the new changes are deployed to the staging. And the QA team takes over the testing part. But when you are in small team, all these has to be achieved with the minimal team. So before the new change is completely pushed to staging, i decided to have a simple testing of all the components quickly. I read about blogs where many DevOps engineers spins up new instances like a full replica of their entire architecture and performs the new code deployment and load test on this new cluster and if all the components are behaving properly with the new code change, it’s then further deployed to Staging for next level of full scale QA.

Though the above step seems to be interesting, i didn’t want to waste up resources by spinnig up a new set of instances each time. Being a hardcore Docker fan, i decided to replace the instance lauch iwth Docker containers. So instead of launching ne instances, Jenkins will launch new Docker containers with SDN(Software Defined Network). Below is simple architecture diagram of my new design.

So the work flow goes like this,

1) Developers pushes the new code changes along with the new Tag to the corresponding Repositories.

2) Github webhook then triggers jenkins to start the Build jobs.

3) Jenkins performs the build and if the build succeeds, jenkins triggers Debian pacakging for the application.

4) Once the packaging is completed, Jenkins will trigger Docker image creation for the corresponding application using the newly build packages.

5) Once the image build is completed, Jenkins uses Docker Compose to build our Virtual clusters which is an exact replica of our Prod/Staging.

6) Once the cluster is up, we perform automated testing of all our components and makes sure that the components are behaving normally with the new code changes.

Now once the test results are normal, we can initiate the code deployment to staging and can start the full scale QA.

By using Docker, i was able to reduce the resource usage. All these containers are running on a Single M3.Medium box. Sice i’m concentrating more on the components working part and not on the load test side, with this smaller box i was able to achieve my results properly.

A bit about docker-compose. I’m using docker-compose for managing the docker cluster. Compose is a tool for defining and running complex applications with Docker. With Compose, we can define a multi-container application in a single file, then spin our applications up in a single command which does everything that needs to be done to get it running. Below is my docker-compose yml file content.

  web:
    image:        web:latest
    links:
      - redis
    ports:
      - "8080:80"
    environment:
      - ENV1
          - ENV2
  redis:
    image:        my_redis:latest
    ports:
      - "172.16.16.17:6379:6379"
  backend:
    image:        my_backend:latest
    net:          "host"

From the initial test results, i was very much satisfied. Now i’m planning to extend this setup to next level including a fully automated load test.

Standard
pkgr

Packaging Node/Python App Using Pkgr

pkgr is a tool for building deb/rpm packages for Python/Ruby/Node/GO applications. It uses heroku buildpack and embed all the dependencies related to the application runtime within the package. It also gives us a nice executable, which closely replicates the Heroku toolbelt utility. There are only 2 requirements for pkgr, 1) It must have a Procfile and 2) It should be Heroku compatible.

By default, pkgr supports packaging Ruby/GO/Node apps. But it also supports custom buildpacks, so we can use heroku-python build pack to pacakge Python apps too.

Installing pkgr

$ apt-get update

$ apt-get install -y build-essential ruby1.9.1-full rubygems1.9.1

$ gem install pkgr

Packaging a Node application

For pacakging a Node application, run the below command

$ pkgr package <path-to-node-app-source> --verbose --debug --env "HOME=/tmp" --auto

Packaging a Python application,

For pacakging a Python application, run the below command

$ pkgr package <path-to-python-app-source> --verbose --debug --env HOME=/tmp --auto --buildpack=https://github.com/heroku/heroku-buildpack-python

Note: python buld pack, we need to have libssl0.9.8 installed, other wise pip install will throw hashlib errors.

Standard
mongodb

Promoting a MongoDB Slave

MongoDB is one of the commonly used NOSQL document store. For smaller use cases, we might not need a full scaled replica set, instead we can use MongoDB in a traditional way like a Master-Slave architecture. In this blog, i’m going to explain how to convert a Standalone MongoDB server to a Master-Slave Model, and Promoting a Slave instance into a Master node in case of master crash.

Standalone to Master-slave Model.

First, on the master node, we need to add master=true on to the mongodb config file and restart the mongo service. On the new mongo node, which is going to be the slave, add the below config options to the mongodb configuration file.

slave=true
source=xxx.xxx.xxx.xxx:yyyyy        # replace with Mongo Master IP:PORT
autoresync=true

Now restart the mongo service on the slave node and tail the mongo logs, we can see the replication info on it. Below is a sample replication details that can be seen on the mongo logs.

2015-04-18T05:09:41.800+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxxx" }
2015-04-18T05:09:41.801+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxxx" }
2015-04-18T05:09:41.801+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxx" }
2015-04-18T05:09:41.802+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxx" }
2015-04-18T05:09:41.802+0000 I REPL     [replslave] resync: done with initial clone for db: testdb
2015-04-18T05:09:51.135+0000 I REPL     [replslave] repl:   applied 1 operations
2015-04-18T05:09:51.135+0000 I REPL     [replslave] repl:  end sync_pullOpLog syncedTo: Apr 18 05:15:40 5531e87c:1
2015-04-18T05:09:51.135+0000 I REPL     [replslave] repl: sleep 1 sec before next pass
2015-04-18T05:09:52.135+0000 I REPL     [replslave] repl: syncing from host:xxx.xxx.xxx.xxx:yyyyy
2015-04-18T05:10:01.135+0000 I REPL     [replslave] repl:   applied 1 operations
2015-04-18T05:10:01.135+0000 I REPL     [replslave] repl:  end sync_pullOpLog syncedTo: Apr 18 05:15:50 5531e886:1
2015-04-18T05:10:01.135+0000 I REPL     [replslave] repl: syncing from host:xxx.xxx.xxx.xxx:yyyyy
2015-04-18T05:10:11.135+0000 I REPL     [replslave] repl:   applied 1 operations

We can also check the replication status from the Mongo master cli via rs.printReplicationInfo() or db.serverStatus( { repl: 1 } ). We can also check the same on the slave nodes, but by default, read queries are not allowed on the slave and it will throw an error. We can allow reads by running db.getMongo().setSlaveOk() on the slave mongo shell. This will override the restriction and we can use the rs.printReplicationInfo() or db.serverStatus( { repl: 1 } ) to see the replication status.

Promoting a Slave node to Master

This is one the requirement that we keep slave nodes. In case of Master crash, we can easily promote the Slave node and can minimize the interruption. Now promoting a Slave node to Master, follow the below steps.

1) Stop the mongo service on the slave

2) remove all the local files from the mongo data directory

$ cd <mongo_data_directory> && rm -rvf local*

3) Remove the slave configurations from mongo config file, and set `master=true` (This is required if we have more than 1 Slaves, so that the rest of the slaves can connect to new master).

4) Restart the mongo service, now this new master ready to accept writes.

If we have multiple slaves, we need to change the slave source IP, so that they can connect to the new master. But even if the connect to the new master, replication will fail. So we have two methods, either remove the data and perform a new data replication or use force a complete resync to all the slaves using the below command

#On the mongo master shell, run

$ use admin

$ db.runCommand( { resync: 1 }      # This will force a complete resync on all the slaves.

This procedure is useful, if you are using a Standalone/Master-Slave method. For a real HA/Fault tolerant design, replica set proves to be more efficient, where primary master selection takes place automatically if the actual primary node crashes, thus preventing the down time to minimum.

Standard