Containers, Microservices are the hottest topics in the IT world now. Container technology like LXC, Docker, rkt etc are being used heavily in production now. It's easy to use these technologies on stateless services, so even if a container is crashed, a new container can be launched within seconds and can start serving traffic. Containers are widely used for serving web traffics. Tools like Marathon, provides a HA-Proxy based service discovery, wherein it reloads the HA-Proxy service based on the start/stop of the containers in the Mesos cluster. A simplest similar method is to use etcD with Docker and Haproxy.
With this we can apply the same logic here also ie, when a container is launched, we can save the IP and Port that is being mapped from the HOST machine and the same can be updated dynamically to HA-Proxy config file using ConfD and reload the HA-Proxy service. Now imagine if a container is terminated due to some reason, the HA-Proxy host check will fail, but the entry will be still there in the haproxy.cfg. Since we will be launching new containers either manually or using frameworks like Marathon/Helios etc, there is no need to to keep the old entries in the haproxy.cfg as well as in the ETCD.
But we need a way to keep track of the containers which got terminated, so that we can remove those entries. Enter serf, Serf is a decentralized solution for cluster membership, failure detection, and orchestration. Serf relies on an efficient and lightweight gossip protocol to communicate with nodes. The Serf agents periodically exchange messages with each other in much the same way that a zombie apocalypse would occur: it starts with one zombie but soon infects everyone. Serf is able to quickly detect failed members and notify the rest of the cluster. This failure detection is built into the heart of the gossip protocol used by Serf.
We will be running Serf agents on each container. These serf clusters are isolated ie, we can control the joining members, thereby maintaining separate serf clusters. So if a Container goes down, other agents will receive this member-failed event, and we can define an event handler that can preform the key removal from the etcD which in turn invoke the confD. Then ConfD will remove the server entry from the HA-Proxy config file.
Now let's see this in action ;), i'm using Ubuntu 14.04 as the Base Host
Setting up HA-Proxy and Docker Web Containers
Let's install HA-Proxy on the base Host,
$ apt-get install haproxy
Enable the Ha-Proxy service in /etc/default/haproxy and start the service
$ /etc/init.d/haproxy start
Now install Docker container,
$ curl -sSL https://get.docker.com/ | sh
Once Docker is installed, lets pull an Ubuntu image from the Docker Registry
$ docker pull ubuntu:14.04
Setting up ETCD
Download the latest version of EtcD
$ curl -L https://github.com/coreos/etcd/releases/download/v2.1.3/etcd-v2.1.3-linux-amd64.tar.gz -o etcd-v2.1.3-linux-amd64.tar.gz
$ tar xvzf etcd-v2.1.3-linux-amd64.tar.gz
Let's start the EtcD services,
$ ./etcd -name default --listen-peer-urls "http://172.17.42.1:2380,http://172.17.42.1:7001" --listen-client-urls "http://172.17.42.1:2379,http://172.17.42.1:4001" --advertise-client-urls "http://172.17.42.1:2379,http://172.17.42.1:4001" --initial-advertise-peer-urls "http://172.17.42.1:2380,http://172.17.42.1:7001" --initial-cluster 'default=http://172.17.42.1:2380,default=http://172.17.42.1:7001'
where 172.17.42.1 is my Host machine IP
Setting up ConfD
Let's setup the ConfD service. For ConfD, we basically need two files, 1) a toml file which contains the info like which keys to lookup in etcd, which etcd cluster lookup, path to the template file etc… and 2) a tmpl (template) file
Download the latest version of confd
$ wget https://github.com/kelseyhightower/confd/releases/download/v0.10.0/confd-0.10.0-linux-amd64
$ cp -rvf confd-0.10.0-linux-amd64 /usr/local/bin/
$ mkdir -p /etc/confd/{conf.d,templates}
Now lets create the tmpl and toml file
toml: /etc/confd/conf.d/myconfig.toml
[template]
src = "haproxy.cfg.tmpl"
dest = "/etc/haproxy/haproxy.cfg"
keys = [
"/proxy/frontend",
]
reload_cmd = "/etc/init.d/haproxy reload"
tmpl: /etc/confd/templates/haproxy.cfg.tmpl
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
contimeout 5000
clitimeout 50000
srvtimeout 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend localnodes
bind *:80
mode http
default_backend nodes
backend nodes
mode http
balance roundrobin
option forwardfor
{{range gets "/proxy/frontend/*"}}
server {{base .Key}} {{.Value}} check
{{end}}
Start the service in foreground by running the below command,
$ confd -backend etcd -node 172.17.42.1:4001 --log-level="debug" -interval=10 # confd will perform lookup for every 10sec
For now, there is no key present in the ETCD, ConfD will perform md5sum lookup of the current file and expected state of the file, if there is any change it will perform updating the file. Below are the debug logs from the ConfD service
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: INFO Backend set to etcd
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: INFO Starting confd
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: INFO Backend nodes set to 172.17.42.1:4001
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: DEBUG Loading template resources from confdir /etc/confd
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: DEBUG Loading template resource from /etc/confd/conf.d/myconfig.toml
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: DEBUG Retrieving keys from store
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: DEBUG Key prefix set to /
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: DEBUG Using source template /etc/confd/templates/haproxy.cfg.tmpl
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: DEBUG Compiling source template /etc/confd/templates/haproxy.cfg.tmpl
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: DEBUG Comparing candidate config to /etc/haproxy/haproxy.cfg
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: INFO /etc/haproxy/haproxy.cfg has md5sum aa00603556cb147c532d5d97f90aaa17 should be fadab72f5cef00c13855a27893d6e39c
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: INFO Target config /etc/haproxy/haproxy.cfg out of sync
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: DEBUG Overwriting target config /etc/haproxy/haproxy.cfg
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: DEBUG Running /etc/init.d/haproxy reload
2015-09-07T07:09:48Z vagrant-ubuntu-trusty-64 ./confd[14378]: DEBUG " * Reloading haproxy haproxy\n ...done.\n"
Configure the Docker Web Containers
Since in the end, our Containers will be running on a different host, HA-Proxy needs to know the HOST IP where the containers resides. If the host IP has a Public IP we can easily perform a lookup and get the public ip, in my current scenario, im running in a Vagrant VM, so i'll be passing the HOST IP via Docker ENV variable during startup. If we do a strict port mapping like 80:80 or 443:443, we wont be able to run more than one Web container in a single HOST. So if we go with random ports, our HA-Proxy needs to know these random ports. These port info will be stored in the ETCD key store, so ConfD can utilize the same.
let's start our First container,
$ docker run --rm=true -it -p 9001:80 -p 8010:7373 -p 8011:5000 -h=web1 -e "FD_IP=192.168.33.102" -e "FD_PORT=9001" -e "SERF_RPC_PORT=8010" -e "SERF_PORT=8011" ubuntu:apache
Where port 9000 is mapped to containers port 80, and port 8010 and 8011 for the Serf Clients, FD_IP is the host IP
Now, lets setup the services for the container
$ apt-get update && apt-get install apache2
$ wget https://dl.bintray.com/mitchellh/serf/0.6.4_linux_amd64.zip
$ unzip 0.6.4_linux_amd64.zip
Let's check the serf in foreground on a screen/tmux
$ ./serf agent -node=web2 -bind=0.0.0.0:5000 -rpc-addr=0.0.0.0:7373 -log-level=debug
If the Serf agent is starting fine, then lets go ahead and create an event handler for the event member-failed, which happens when a container is crashed. Below is a simple event handler script
#serf_handler.py
#!/usr/bin/python
import sys
import os
import requests
import json
base_url = "http://172.17.42.1:4001" # Use any config mgmt tool or user-data script to populate this value for each host
base_key = "/v2/keys"
cluster_key = ["/proxy/frontend/", "/proxy/serf/", "/proxy/serf-rpc/"] # EtcD keys that has to be removed
for line in sys.stdin:
event_type = os.environ['SERF_EVENT']
if event_type == 'member-failed':
address = line.split('\t')[0]
for key in cluster_key:
full_url = base_url + base_key + key + address
print full_url
r = requests.get(full_url)
if r.status_code == 200:
r = requests.delete(full_url)
print "Key Successfully removed from ETCD"
if r.status_code == 404:
print "Key already removed by another Serf Agent"
Also we need a bootstrapping script that can add the necessary keys onto ETCD, when the container has started for the first time. This script has to be invoked every during the starting of the container, so ConfD is aware of the new Containers that have started. Below is a simple shell script for the same,
#etcd_bootstrap.sh
HOST=`echo $HOSTNAME`
ETCD_VAL=`echo "$FD_IP:$FD_PORT"`
ETCD_SERF_PORT=`echo $FD_IP:$SERF_PORT`
ETCD_SERF_RPC_PORT=`echo $FD_IP:$SERF_RPC_PORT`
curl -s -X PUT "http://172.17.42.1:4001/v2/keys/proxy/frontend/$HOST" -d value=$ETCD_VAL > /dev/null
exit_stat=`echo $?`
if [ $exit_stat == 0 ]; then
curl -s -X POST --data-urlencode 'payload={"channel": "#docker", "username": "etcdbot", "text": "`Host: '"$HOST"'`, FD key has been added successfully to ETCD cluster", "icon_emoji": ":ghost:"}' https://hooks.slack.com/services/xxxxxxx/yyyyyyy/xyxyxyxyxyxyxyxyxyxyxyx
else
curl -s -X POST --data-urlencode 'payload={"channel": "#docker", "username": "etcdbot", "text": "`Host: '"$HOST"'`, Failed to add FD key to the ETCD cluster", "icon_emoji": ":ghost:"}' https://hooks.slack.com/services/xxxxxxx/yyyyyyy/xyxyxyxyxyxyxyxyxyxyxyx
fi
curl -s -X PUT "http://172.17.42.1:4001/v2/keys/proxy/serf/$HOST" -d value=$ETCD_SERF_PORT > /dev/null
exit_stat=`echo $?`
if [ $exit_stat == 0 ]; then
curl -s -X POST --data-urlencode 'payload={"channel": "#docker", "username": "etcdbot", "text": "`Host: '"$HOST"'`, SERF PORT key has been added successfully to ETCD cluster", "icon_emoji": ":ghost:"}' https://hooks.slack.com/services/xxxxxxx/yyyyyyy/xyxyxyxyxyxyxyxyxyxyxyx
else
curl -s -X POST --data-urlencode 'payload={"channel": "#docker", "username": "etcdbot", "text": "`Host: '"$HOST"'`, Failed to add SERF PORT key to the ETCD cluster", "icon_emoji": ":ghost:"}' https://hooks.slack.com/services/xxxxxxx/yyyyyyy/xyxyxyxyxyxyxyxyxyxyxyx
fi
curl -s -X PUT "http://172.17.42.1:4001/v2/keys/proxy/serf-rpc/$HOST" -d value=$ETCD_SERF_RPC_PORT > /dev/null
exit_stat=`echo $?`
if [ $exit_stat == 0 ]; then
curl -s -X POST --data-urlencode 'payload={"channel": "#docker", "username": "etcdbot", "text": "`Host: '"$HOST"'`, SERF RPC PORT key has been added successfully to ETCD cluster", "icon_emoji": ":ghost:"}' https://hooks.slack.com/services/xxxxxxx/yyyyyyy/xyxyxyxyxyxyxyxyxyxyxyx
else
curl -s -X POST --data-urlencode 'payload={"channel": "#docker", "username": "etcdbot", "text": "`Host: '"$HOST"'`, Failed to add SERF RPC PORT key to the ETCD cluster", "icon_emoji": ":ghost:"}' https://hooks.slack.com/services/xxxxxxx/yyyyyyy/xyxyxyxyxyxyxyxyxyxyxyx
fi
Now lets add the necessary keys to ETCD,
$ bash /opt/scripts/etcd_bootstrap.sh
Once the keys are added, lets start the serf agent wit the event handler script,
$ ./serf agent -node=web1 -bind=0.0.0.0:5000 -rpc-addr=0.0.0.0:7373 -log-level=debug -event-handler=/opt/scripts/serf_handler.py
# Script output
==> Starting Serf agent...
==> Starting Serf agent RPC...
== > Serf agent running!
Node name: 'web1'
Bind addr: '0.0.0.0:5000'
RPC addr: '0.0.0.0:7373'
Encrypted: false
Snapshot: false
Profile: lan
==> Log data will now stream in as it occurs:
2015/09/07 07:37:12 [INFO] agent: Serf agent starting
2015/09/07 07:37:12 [INFO] serf: EventMemberJoin: web1 172.17.0.20
2015/09/07 07:37:13 [INFO] agent: Received event: member-join
Now we have a running Web Container, since we have added the keys to ETCD, lets verify that ConfD has detected the changes. Below is the logs from the ConfD,
2015-09-07T07:40:17Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Retrieving keys from store
2015-09-07T07:40:17Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Key prefix set to /
2015-09-07T07:40:17Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Using source template /etc/confd/templates/haproxy.cfg.tmpl
2015-09-07T07:40:17Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Compiling source template /etc/confd/templates/haproxy.cfg.tmpl
2015-09-07T07:40:17Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Comparing candidate config to /etc/haproxy/haproxy.cfg
2015-09-07T07:40:17Z vagrant-ubuntu-trusty-64 ./confd[14970]: INFO /etc/haproxy/haproxy.cfg has md5sum fe4fbaa3f5782c7738a365010a5f6d48 should be fadab72f5cef00c13855a27893d6e39c
2015-09-07T07:40:17Z vagrant-ubuntu-trusty-64 ./confd[14970]: INFO Target config /etc/haproxy/haproxy.cfg out of sync
2015-09-07T07:40:17Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Overwriting target config /etc/haproxy/haproxy.cfg
2015-09-07T07:40:17Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Running /etc/init.d/haproxy reload
2015-09-07T07:40:17Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG " * Reloading haproxy haproxy\n ...done.\n"
2015-09-07T07:40:17Z vagrant-ubuntu-trusty-64 ./confd[14970]: INFO Target config /etc/haproxy/haproxy.cfg has been updated
Also, lets confirm that the haproxy.cfg file has been updated,
# Updated part from the file
backend nodes
mode http
balance roundrobin
option forwardfor
server web1 192.168.33.102:9000 check # ConfD has updated the server entry based on the key present in the EtcD
Now let's add the second Web container using the steps we followed for the first container, make sure that the ports are updated on the run script (For the second container, im using 9001 as the web port, 8020/8021 as the Serf ports). Once the bootstrapping scrip is executed successfully, lets verify if the haproxy.cfg file is update with the second entry.
backend nodes
mode http
balance roundrobin
option forwardfor
server web1 192.168.33.102:9000 check
server web2 192.168.33.102:9001 check
For now, the serf agents on both the containers are running as a standalone service and they are not aware of each other. Lets execute serf join command to join the cluster. Below is a simple python script for the same.
# serf_starter.py
#!/usr/bin/python
import sys
import os
import etcd
import random
import subprocess
node_keys = []
client = etcd.Client(host='172.17.42.1', port=4001)
r = client.read('/proxy/serf-rpc', recursive = True)
for child in r.children:
node_keys.append(child.key)
node_key = (random.choice(node_keys)) # Pick one server randomly
remote_serf = client.read(node_key)
print remote_serf.value
local_serf = os.environ['FD_IP'] + ":" + os.environ['SERF_PORT']
serf_args = "-rpc-addr=%s %s" %(remote_serf.value, local_serf)
print subprocess.call(["/serf", "join", "-rpc-addr=%s" %remote_serf.value, local_serf])
Run the join script on the second container and check for the events on the first container's serf agent logs
$ /opt/scripts/serf_starter.py
Below are the event logs on the first container,
2015/09/07 07:53:02 [INFO] agent.ipc: Accepted client: 172.17.42.1:37326 # Connection from Web2
2015/09/07 07:53:02 [INFO] agent: joining: [192.168.33.102:8021] replay: false
2015/09/07 07:53:02 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.33.102:8021
2015/09/07 07:53:02 [INFO] serf: EventMemberJoin: web2 172.17.0.35
2015/09/07 07:53:02 [INFO] agent: joined: 1 nodes
2015/09/07 07:53:02 [DEBUG] serf: messageJoinType: web2
Now both the agents are connected, lets verify the same
./serf members # we can run this command on any one of the containers
web2 172.17.0.35:5000 alive
web1 172.17.0.20:5000 alive
Our Serf Cluster is up and running. Now lets terminate one container and see if Serf is able to detect the event. Once the serf detects the event it will call the event_handler script which should remove the keys from the etcD. In this case, i'm terminating the second container. We could see the event popping up in the first container's serf agent logs.
2015/09/07 07:57:37 [INFO] memberlist: Suspect web2 has failed, no acks received
2015/09/07 07:57:39 [INFO] memberlist: Suspect web2 has failed, no acks received
2015/09/07 07:57:41 [INFO] memberlist: Suspect web2 has failed, no acks received
2015/09/07 07:57:42 [INFO] memberlist: Suspect web2 has failed, no acks received
2015/09/07 07:57:42 [INFO] memberlist: Marking web2 as failed, suspect timeout reached
2015/09/07 07:57:42 [INFO] serf: EventMemberFailed: web2 172.17.0.35
2015/09/07 07:57:42 [INFO] serf: attempting reconnect to web2 172.17.0.35:5000
2015/09/07 07:57:43 [INFO] agent: Received event: member-failed
2015/09/07 07:57:43 [DEBUG] agent: Event 'member-failed' script output: http://172.17.42.1:4001/v2/keys/proxy/frontend/web2
Key Successfully removed from ETCD
http://172.17.42.1:4001/v2/keys/proxy/serf/web2
Key Successfully removed from ETCD
http://172.17.42.1:4001/v2/keys/proxy/serf-rpc/web2
Key Successfully removed from ETCD
As per the event_handler script output, the key has been successfully removed from the ETCD. Lets check the confd logs
# Confd logs
2015-09-07T07:57:47Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Retrieving keys from store
2015-09-07T07:57:47Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Key prefix set to /
2015-09-07T07:57:47Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Using source template /etc/confd/templates/haproxy.cfg.tmpl
2015-09-07T07:57:47Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Compiling source template /etc/confd/templates/haproxy.cfg.tmpl
2015-09-07T07:57:47Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Comparing candidate config to /etc/haproxy/haproxy.cfg
2015-09-07T07:57:47Z vagrant-ubuntu-trusty-64 ./confd[14970]: INFO /etc/haproxy/haproxy.cfg has md5sum fadab72f5cef00c13855a27893d6e39c should be df9f09eb110366f2ecfa43964a3c862d
2015-09-07T07:57:47Z vagrant-ubuntu-trusty-64 ./confd[14970]: INFO Target config /etc/haproxy/haproxy.cfg out of sync
2015-09-07T07:57:47Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Overwriting target config /etc/haproxy/haproxy.cfg
2015-09-07T07:57:47Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG Running /etc/init.d/haproxy reload
2015-09-07T07:57:47Z vagrant-ubuntu-trusty-64 ./confd[14970]: DEBUG " * Reloading haproxy haproxy\n ...done.\n"
2015-09-07T07:57:47Z vagrant-ubuntu-trusty-64 ./confd[14970]: INFO Target config /etc/haproxy/haproxy.cfg has been updated
# haproxy.cfg file
backend nodes
mode http
balance roundrobin
option forwardfor
server web1 192.168.33.102:9000 check
Bingo 🙂 , confd has detected the key removal and it has updated the haproxy.cfg file :). We can even remove the bootstrap script and can use Serf event handlers to populate the key when the member-join event is triggered
With ConfD, Serf, ETCD we can keep our service config files up to date with any human intervention. No matter if we scale up or scale down our Docker containers, our system will get updated automatically :). Put an option to log the events onto Slack and that's it, we wont miss any events and our whole team can keep track of what's happening under the hood
beingasysadmin 