Ansible, Redis

Building an Automated Config Management Server using Ansible+Flask+Redis

It’s almost 2 months since i’ve started playing full time on ansible. Like most of the SYS-Admin’s, ive been using ansible via cli most of the time. Unlike Salt/Puppet, ansible is an agent less one. So we need to invoke things from the box which contains ansible and the respective playbooks installed. Also, if you want to use ansible with ec2 features like auto-scaling, we need to either buy Ansible Tower, or need to use ansible-fetch along with the userdata script. I’ve also seen people, who uses custom scripts, that fetches their repo and execute ansible playbook locally to bootstrap.

Being a good fan of Flask, i’ve used flask on creating many backend API’s to automate a bunch of my tasks. So this time i decided to write a simple Flask API for executing Ansible playbook/ Ansible Adhoc commands etc.. Ansible also provides a Python API, which also made my work easier. Like most of the Ansible user’s, i use Role’s for all my playbooks. We can directly expose an API to ansible and can execute playbooks. But there are cases, where the playbook execution takes > 5min, and offcourse if there is any network latency it will affect our package download etc. I don’t want to force my HTTP clients to wait for the final output of the playbook execution to get a response back.

So i decided to go ahead with a JOB Queue feature. Each time a request comes to my API, the job is queued in Redis and the JOB ID will be returned to the clients. Then my job-workers pick the job’s from the redis queue and performs the job execution on the backend and workers will keep updating the job status. So now, i need to expose 2 API’s first, ie, one for receiving jobs and one for job status. For Redis Queue, there is an awesome library called rq. I’ve been using rq for all queuing tasks.

Flask API

The JOB accepts a bunch of parameters like host, role, env via HTTP POST method. Since the role/host etc.. have to be retrieved from the HTTP request, my playbook yml file has to be a dynamic one. So i’ve decided to use Jinja templating to dynamically create my playbook yml file. Below is my sample API for Role based playbook execution.

@app.route('/ansible/role/', methods=['POST'])
def role():
  inst_ip = request.form['host']                          # Host to which the playbook has to be executed
  inst_role = request.form['role']                        # Role to be applied on the Playbook
  env = request.form['env']               # Extra evn variables to be passed while executing the playbook
  ans_remote_user = "ubuntu"                  # Default remote user
  ans_private_key = "/home/ubuntu/.ssh/id_rsa"        # Default ssh private key
  job = q.enqueue_call(                   # Queuing the job on to Redis
            func=ansble_run, args=(inst_ip, inst_role, env, ans_remote_user, ans_private_key,), result_ttl=5000, timeout=2000
  return job.get_id()                     # Returns job id if the job is successfully queued to Redis

Below is a sample templating function that generates the playbook yml file via Jinja2 templating

def gen_pbook_yml(ip, role):
  r_text = ''
  templateLoader = jinja2.FileSystemLoader( searchpath="/" )
  templateEnv = jinja2.Environment( loader=templateLoader )
  TEMPLATE_FILE = "/opt/ansible/playbook.jinja"                # Jinja template file location
  template = templateEnv.get_template( TEMPLATE_FILE )
  role = role.split(',')                       # Make Role as an array if Multiple Roles are mentioned in the POST request
  r_text = ''.join([random.choice(string.ascii_letters + string.digits) for n in xrange(32)])  
  temp_file = "/tmp/" + "ans-" + r_text + ".yml"           # Crating a unique playbook yml file
  templateVars = { "hst": ip,
                   "roles": role
  outputText = template.render( templateVars )             # Rendering template
  text_file = open(temp_file, "w")
  text_file.write(outputText)                      # Saving the template output to the temp file
  return temp_file

Once the playbook file is ready, we need to invoke Ansible’s API to perform our bootstrapping. This is actually done by the Job workers. Below is a sample function which invokes the playbook API from Ansible CORE.

def ansble_run(ans_inst_ip, ans_inst_role, ans_env, ans_user, ans_key_file):
  yml_pbook = gen_pbook_yml(ans_inst_ip, ans_inst_role)   # Generating the playbook yml file
  run_pbook = ansible.playbook.PlayBook(          # Invoking Ansible's playbook API
                 host_list="/etc/ansible/hosts",          # use either host_file or inventory
#                Inventory='path/to/inventory/file',
                    'env': ans_env
  return run_pbook                    # We can tune the output that has to be returned

Now the job-workers executes and updates the status on the Redis. Now we need to expose our JOB status API. Below is a sample Flask API for the same.

@app.route("/ansible/results/<job_key>", methods=['GET'])
def get_results(job_key):

    job = Job.fetch(job_key, connection=conn)
    if job.is_finished:
        ret = job.return_value
    elif job.is_queued:
        ret = {'status':'in-queue'}
    elif job.is_started:
        ret = {'status':'waiting'}
    elif job.is_failed:
        ret = {'status': 'failed'}

    return json.dumps(ret), 200

Now, we have a fully fledged API server for executing Role based playbooks. This API can also be used with user data scripts in autoscaling, where in we need to perform an HTTP POST request to the API server, and our API server will start the Bootstrapping. I’ve tested this app locally with various scenarios and the results are promising. Now as a next step, i’m planning to extend the API to do more jobs like, automating Code Pushes, Running AD-Hoc commands via API etc… With applications like Ansible, Redis, Flask, i’m sure SYS Admins can attain the DevOps Nirvana :). I’ll be pushing the latest working code to my Github account soon…

Docker, Jenkins

Virtual Cluster Testing Using Jenkins and Docker

Nowadays CI or Conitnous Integration is being implemented in almost all IT companies. Many of the DevOps work’s are in related to the CI. The common scenario is, Developers push the codes to the GIT/SVN repo and triggers jenkins to perform tests and sometimes packaging, and if it’s a fuly automated system the new changes are deployed to the staging. And the QA team takes over the testing part. But when you are in small team, all these has to be achieved with the minimal team. So before the new change is completely pushed to staging, i decided to have a simple testing of all the components quickly. I read about blogs where many DevOps engineers spins up new instances like a full replica of their entire architecture and performs the new code deployment and load test on this new cluster and if all the components are behaving properly with the new code change, it’s then further deployed to Staging for next level of full scale QA.

Though the above step seems to be interesting, i didn’t want to waste up resources by spinnig up a new set of instances each time. Being a hardcore Docker fan, i decided to replace the instance lauch iwth Docker containers. So instead of launching ne instances, Jenkins will launch new Docker containers with SDN(Software Defined Network). Below is simple architecture diagram of my new design.

So the work flow goes like this,

1) Developers pushes the new code changes along with the new Tag to the corresponding Repositories.

2) Github webhook then triggers jenkins to start the Build jobs.

3) Jenkins performs the build and if the build succeeds, jenkins triggers Debian pacakging for the application.

4) Once the packaging is completed, Jenkins will trigger Docker image creation for the corresponding application using the newly build packages.

5) Once the image build is completed, Jenkins uses Docker Compose to build our Virtual clusters which is an exact replica of our Prod/Staging.

6) Once the cluster is up, we perform automated testing of all our components and makes sure that the components are behaving normally with the new code changes.

Now once the test results are normal, we can initiate the code deployment to staging and can start the full scale QA.

By using Docker, i was able to reduce the resource usage. All these containers are running on a Single M3.Medium box. Sice i’m concentrating more on the components working part and not on the load test side, with this smaller box i was able to achieve my results properly.

A bit about docker-compose. I’m using docker-compose for managing the docker cluster. Compose is a tool for defining and running complex applications with Docker. With Compose, we can define a multi-container application in a single file, then spin our applications up in a single command which does everything that needs to be done to get it running. Below is my docker-compose yml file content.

    image:        web:latest
      - redis
      - "8080:80"
      - ENV1
          - ENV2
    image:        my_redis:latest
      - ""
    image:        my_backend:latest
    net:          "host"

From the initial test results, i was very much satisfied. Now i’m planning to extend this setup to next level including a fully automated load test.


Packaging Node/Python App Using Pkgr

pkgr is a tool for building deb/rpm packages for Python/Ruby/Node/GO applications. It uses heroku buildpack and embed all the dependencies related to the application runtime within the package. It also gives us a nice executable, which closely replicates the Heroku toolbelt utility. There are only 2 requirements for pkgr, 1) It must have a Procfile and 2) It should be Heroku compatible.

By default, pkgr supports packaging Ruby/GO/Node apps. But it also supports custom buildpacks, so we can use heroku-python build pack to pacakge Python apps too.

Installing pkgr

$ apt-get update

$ apt-get install -y build-essential ruby1.9.1-full rubygems1.9.1

$ gem install pkgr

Packaging a Node application

For pacakging a Node application, run the below command

$ pkgr package <path-to-node-app-source> --verbose --debug --env "HOME=/tmp" --auto

Packaging a Python application,

For pacakging a Python application, run the below command

$ pkgr package <path-to-python-app-source> --verbose --debug --env HOME=/tmp --auto --buildpack=

Note: python buld pack, we need to have libssl0.9.8 installed, other wise pip install will throw hashlib errors.


Promoting a MongoDB Slave

MongoDB is one of the commonly used NOSQL document store. For smaller use cases, we might not need a full scaled replica set, instead we can use MongoDB in a traditional way like a Master-Slave architecture. In this blog, i’m going to explain how to convert a Standalone MongoDB server to a Master-Slave Model, and Promoting a Slave instance into a Master node in case of master crash.

Standalone to Master-slave Model.

First, on the master node, we need to add master=true on to the mongodb config file and restart the mongo service. On the new mongo node, which is going to be the slave, add the below config options to the mongodb configuration file.

slave=true        # replace with Mongo Master IP:PORT

Now restart the mongo service on the slave node and tail the mongo logs, we can see the replication info on it. Below is a sample replication details that can be seen on the mongo logs.

2015-04-18T05:09:41.800+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxxx" }
2015-04-18T05:09:41.801+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxxx" }
2015-04-18T05:09:41.801+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxx" }
2015-04-18T05:09:41.802+0000 I STORAGE  [replslave] copying indexes for: { name: "xxxxxxx" }
2015-04-18T05:09:41.802+0000 I REPL     [replslave] resync: done with initial clone for db: testdb
2015-04-18T05:09:51.135+0000 I REPL     [replslave] repl:   applied 1 operations
2015-04-18T05:09:51.135+0000 I REPL     [replslave] repl:  end sync_pullOpLog syncedTo: Apr 18 05:15:40 5531e87c:1
2015-04-18T05:09:51.135+0000 I REPL     [replslave] repl: sleep 1 sec before next pass
2015-04-18T05:09:52.135+0000 I REPL     [replslave] repl: syncing from
2015-04-18T05:10:01.135+0000 I REPL     [replslave] repl:   applied 1 operations
2015-04-18T05:10:01.135+0000 I REPL     [replslave] repl:  end sync_pullOpLog syncedTo: Apr 18 05:15:50 5531e886:1
2015-04-18T05:10:01.135+0000 I REPL     [replslave] repl: syncing from
2015-04-18T05:10:11.135+0000 I REPL     [replslave] repl:   applied 1 operations

We can also check the replication status from the Mongo master cli via rs.printReplicationInfo() or db.serverStatus( { repl: 1 } ). We can also check the same on the slave nodes, but by default, read queries are not allowed on the slave and it will throw an error. We can allow reads by running db.getMongo().setSlaveOk() on the slave mongo shell. This will override the restriction and we can use the rs.printReplicationInfo() or db.serverStatus( { repl: 1 } ) to see the replication status.

Promoting a Slave node to Master

This is one the requirement that we keep slave nodes. In case of Master crash, we can easily promote the Slave node and can minimize the interruption. Now promoting a Slave node to Master, follow the below steps.

1) Stop the mongo service on the slave

2) remove all the local files from the mongo data directory

$ cd <mongo_data_directory> && rm -rvf local*

3) Remove the slave configurations from mongo config file, and set `master=true` (This is required if we have more than 1 Slaves, so that the rest of the slaves can connect to new master).

4) Restart the mongo service, now this new master ready to accept writes.

If we have multiple slaves, we need to change the slave source IP, so that they can connect to the new master. But even if the connect to the new master, replication will fail. So we have two methods, either remove the data and perform a new data replication or use force a complete resync to all the slaves using the below command

#On the mongo master shell, run

$ use admin

$ db.runCommand( { resync: 1 }      # This will force a complete resync on all the slaves.

This procedure is useful, if you are using a Standalone/Master-Slave method. For a real HA/Fault tolerant design, replica set proves to be more efficient, where primary master selection takes place automatically if the actual primary node crashes, thus preventing the down time to minimum.


Kannel: Open Source SMS Gateway

It’s been quite a while since my last blog. This time i’m coming with a bunch of topics to write, starting with Kannel. After moving to my new role, the first task i got was to set up an SMPP server with one of our carriers. After digging sometime in internet i found one project kannel, which is a perfect game player for me. So in this blog, i’ll be explaining on how to setup an SMPP SMS gateway locally.

Installing Kannel

Download the latest source code from kannel site.

$ wget -O /opt/

$ cd /opt && tar xvzf kannel-snapshot.tar.gz && cd kannel-snapshot

$ apt-get install -y libxml2-dev libxml2 openssl libssl-dev build-essential      # installing dependencies

$ ./configure --prefix=/usr/local/kannel/

$ make && make install

$ adduser --system --home /usr/local/kannel/lib/kannel/ --no-create-home --gecos "Kannel" kannel

$ mkdir /var/log/kannel && mkdir /var/run/kannel

$ chown -c kannel.root /var/log/kannel && chown -c kannel.root /var/run/kannel 

Now we have the kannel installed on our custom prefix folder. Let’s go ahead setting the Kannel application.

Setting up Kannel

Kannel comprises of two processes, smsbox and bearerbox. Bearerbox service is the one which is in contact with the carrier gateways, responsible for sending and receiving SMS. smsbox is the service which interacts between our application and bearerbox. ie, it receives incoming sms from our bearer box and sends it our application and vice versa. The kannel config consists of multiple parts, which are explained below.

1) Basic configuration: We define the basic details like, bindip, log file path, adminUI port, adminUI password, whitelist ip for accessing admin ui, smsbox port etc..

# sample configuration
group = core
admin-port = 13000
smsbox-port = 13001
admin-password = changeme
admin-deny-ip = "*.*.*.*"
admin-allow-ip = ""
wdp-interface-name = "*"
log-file = "/var/log/kannel/bearerbox.log"
access-log = "/var/log/kannel/access.log"
box-deny-ip = "*.*.*.*"
box-allow-ip = ""

2) SMSC configuration:  We define the smmp details of our carrier's. which includes carrier's smpp ip, smpp port, auth credentials etc..

# sample configuration
group = smsc            # Default group name, no need to modify
smsc = smpp         
host = x.x.x.x
port = yyyy
smsc-id = fake-carrier          # Unique name for this connection
smsc-username = xxxxxxxx
smsc-password = yyyyyyyy

3) smsbox configuration: We define the configurations for the smsbox which includes bindip, a unique id for the smsbox etc..

# sample configuration
group = smsbox
bearerbox-host = localhost
sendsms-interface =
smsbox-id = mysmpp
sendsms-port = 10200                     # Applications make HTTP request to this port
log-file = "/var/log/kannel/smsbox.log"

4) Kannel gateway configuration: We define the user name, password, ratelimit etc for the messages from the smsbox

# sample configuration
group=sendsms-user      # default group name, no need to change
max-messages=3          # sms rate limit

5) SMS service configuration: We define settings for incoming sms from the bearer box, which includes to which URL our application URL to which the SMS details has to be forwarded.

# sample configuration
group = sms-service                 # Default group name
keyword = default
post-url = http://localhost:5000/incoming # When a message is received from SMS center this URL is called. Refer manual for wildcards details.
catch-all = 1
max-messages = 0
omit-empty = true
send-sender = true

Add all the above configurations according to the requirement on to the kannel.conf file. A sample init script for Debian/Ubuntu is available here

Once the SMPP service is started, check the bearebox logs for the connectivity with the carrier’s smpp gateway. Once the connection is up, we can start to send/receive sms. For incoming sms, smsbox will make an HTTP request based on our configuration. For example, if we are using a POST method, the sms details like From, To can be retrieved from the POST HEADERS and the sms text from the request data. Below are some of the headers that come along with the POST requests.

X-Kannel-From   => sender id
X-Kannel-To     => recepient id

Similarly for outbound sms, our application makes a HTTP GET request to the smsbox url and smsbox will carry it over to the bearerbox which then carry over to the carrier for delivery.

Ansible, Ubuntu

Stepping into Ansible

For the past 2 year’s, i played with config management tools like Puppet and Salt. But all these tools were mostly Client-Server Model, except Salt where it supports Push model also. But for the last 6 months, Ansible is gaining more popularity. Ansible is a Push model system which relies on SSH. So before i adopt Ansible completely, i decided to have a try. I need to make sure that the Ansible supports all basic features what other competitors supports. Which is really helpful in migration also.


Ansible is pretty easy to install. We can install it from source or via package managers or even via PIP.We can use the official ubuntu ppa for installing Ansible.

apt-get install software-properties-common
apt-add-repository ppa:ansible/ansible
apt-get update
apt-get install ansible

Since Ansible relies on SSH, things like Host Key verification errors will prevent the SSH connections resulting in failures. We can disable the Host Key Verfication check in the ansible.cfg file

host_key_checking = False      # add this option to the config file

or we can set an env variable export ANSIBLE_HOST_KEY_CHECKING=False for the current session. By default ansible uses the hosts file present in the ansible home directory. So we can define the static machines there. We can add either the IP or DNS resolvable FQDN. Once the IP/FQDN is added, we can test the connectivity via ping module. Make sure that the Ansible server’s SSH key is added to the authorized_keys on the remote machines.

ansible all -m ping

# Sample output

ansible-ubuntu | success >> {
    "changed": false,
    "ping": "pong"

Managing Custom Facts

Config management tools like puppet/Salt supports custom facts to be defined on the remote machines. We can define the custom facts and the config management server can use these facts. Even though Ansible is an agentless server, we can define the custom facts on the remote systems. Whenever we query for facts, ansible connects to the remote machines and fetches the facts using its default library. But it also looks for custom facts in /etc/ansible/facts.d/. We need to put our custom facts file in this directory. The file has to be of .fact extension,must be executable and should return a valid JSON. This is in the case of a script. If we just want to define some facts directly, we can simple create a file like below


The above fact file will add two fact variables called role and profile with the value as mentioned in the file. Now let’s use the system module and see if we are able to retrieve the new custom facts.

ansible <remote_host_name> -m setup

Below is the part of the output showing the custom facts

"ansible_local": {
       "myfacts": {
        "myfact": {
           "profile": "staging",
           "role": "test"

Managing Dynamic Inventory

In the Cloud environment, it’s difficult to maintain a static inventory. Ansible does supports Dynamic inventory for vendors including AWS EC2. Ansible provides us an Inventory script. We can also use this script directly and query EC2 to get the list of all instances. To successfully make an API call to AWS, we will need to configure Boto. The simplest is just to export two environment variables:

export AWS_ACCESS_KEY_ID='AK123'
export AWS_SECRET_ACCESS_KEY='abc123'

./ --list   # Displays the list of all instances

Now we have the inventory script ready. Let’s make some fact’s query to the ec2 instances. We can use many filters here like, Tags etc…

ansible tag_Name_test -i /etc/ansible/plugins/inventory/ -m ping # Querying all instances with tag "Name=test"

We can also use regex with these say like tag_Name_test*. For rackspace user’s there is an official module called rax that works perfectly with ansible

Enrcypting YAML Data files

This is an important feature that most of the config management system lacks. In most of the current systems, we need to define the sensitive data like say ssh-keys, API’s AuthID/Token etc… in plain text which increases the security risk. Ansible Vault comes for rescue here. Vault feature can encrypt any structured data file used by Ansible. This can include “group_vars/” or “host_vars/” inventory variables, variables loaded by “include_vars” or “vars_files”, or variable files passed on the ansible-playbook command line with “-e @file.yml” or “-e @file.json”. Role variables and defaults are also included!. While invoking any playbook, we can pass the --ask-vault-pass along the vault password, so Ansible can can decrypt the file and use its contents while performing any execution.

ansible-vault encrypt foo.yml    # Encrypting a file

ansible-vault edit foo.yml       # Editing encrpypted file

ansible-vault decrypt foo.yml    # Decrypting a file

Ansible indeed is truly an awesome product. It does have many new features like vault compared to its competitors. It’s backed by an awesome community. So we can expect more exciting features in future.


Setting Up Docker Private Registry

Last year Containers based technology showed up big boom. A lot of OpenSource projects and startups wrapped over Docker. Now Docker became a favourite tool for both Dev and Ops guys. I’m a big fan of Docker and i do all my hacks on containers. This time i decided to play with Docker private registry, so that i sync all my docker clients with a central registry. In this test setup i’m using Ubuntu 12.04 server with Nginx as a reverse proxy. With the Nginx proxy i can easily enforce basic auth and can protect my private docker registry from unauthorized access.

Installing Docker Registry

Download the latest release of Docker Registry from the Docker’s github repo

$ wget -O /usr/local/src/0.9.0.tar.gz

$ tar xvzf 0.9.0.tar.gz && mv docker-registry-0.9.0 docker-registry

Let's install the dependencies,

$ apt-get update && apt-get install swig python-pip python-dev libssl-dev liblzma-dev libevent1-dev patch

Once the dependencies are installed, lets go ahead and install the docker-registry app

$ cat /usr/local/src/docker-registry/config/boto.cfg > /etc/boto.cfg

$ pip install /usr/local/src/docker-registry/depends/docker-registry-core/

$ pip install file:///usr/local/src/docker-registry

$ patch $(python -c 'import boto; import os; print os.path.dirname(boto.__file__)')/ < /usr/local/src/docker-registry/contrib/boto_header_patch.diff

$ cp /usr/local/src/docker-registry/config/config_sample.yml /usr/local/src/docker-registry/config/config.yml 

We can edit the `config.yml` file, if we want to change the local storage path (default => /tmp/registry) and also we can use redis as a local cache + sqlite based search backend. The repo already contains a sample init [init]( script that can be used directly.

$ cp -rvf /usr/local/src/docker-registry/contrib/ /etc/init.d/docker-registry

Also let's setup a default file, `/etc/default/docker-registry`, so that init script can read the necessary env variables. Below is the content of my default file.


Let's start the registry service,

$ /etc/init.d/docker-registry start

Setting up Docker Client

Now we have a private Docker registry running, Now let’s setup an Nginx proxy, so that we dont have to expose the registry directly to outside world.

$ apt-get install nginx-extras

The docker-registry repo also contains basic nginx config that can be used directly.

$ cat /usr/local/src/docker-registry/contrib/nginx/nginx.conf > /etc/nginx/sites-enabled/default

$ cp /usr/local/src/docker-registry/contrib/nginx/docker-registry.conf /etc/nginx/

Also create a basic auth file that contains the username password. We can use htpasswd to generate the password. The filename mentioned the nginx config is docker-registry.htpasswd

$ echo "dockeradmin:$apr1$.BzsRrxN$fng.12mJL/TJenKjkZSMS0" >> /etc/nginx/docker-registry.htpasswd  # replace the username and password with the one generated by htpasswd

Now let’s generate a self signed SSL certificate that can be used with nginx. There are websites like StartSSL which provides free 1 year SSL certificate.

$ mkdir /opt/certs && cd /opt/certs

$ openssl genrsa -out devdockerCA.key 2048

$ openssl req -x509 -new -nodes -key devdockerCA.key -days 10000 -out devdockerCA.crt

$ openssl genrsa -out 2048

$ openssl req -new -key -out

    Country Name (2 letter code) [AU]: US
        State or Province Name (full name) [Some-State]: CA
        Locality Name (eg, city) []: SF
        Organization Name (eg, company) [Internet Widgits Pty Ltd]: Beingasysadmin
        Organizational Unit Name (eg, section) []: tech
        Common Name (e.g. server FQDN or YOUR name) []:
        Email Address []:

        Please enter the following 'extra' attributes
        to be sent with your certificate request
        A challenge password []:                          # leave the password blank
        An optional company name []:

$ openssl x509 -req -in -CA devdockerCA.crt -CAkey devdockerCA.key -CAcreateserial -out -days 10000

Copy the certificates to the SSL path mentioned the nginx config,

$ cp /etc/ssl/certs/docker-registry

$ cp /etc/ssl/private/docker-registry

Now let’s restart the nginx process to reflect the changes

$ service nginx restart

Now once the nginx is up, we can check the connectivity between docker client and registry server. Since registry is using a self signed certificate, we need to whitelist the CA on the Docker client machine.

$ cd /opt/certs

$ mkdir /usr/local/share/ca-certificates/docker-dev-cert

$ cp devdockerCA.crt /usr/local/share/ca-certificates/docker-dev-cert

$ update-ca-certificates

Note: If the CA is not added to trusted list, Docker client wont be able to authenticate against the registry server. Once the CA is added to trusted list, we can test the connectivity between Docker client and Registry server. If the Docker daemon was running before adding the CA, then we need to restart the Docker daemon

$ root@docker:~# docker login      # use the nginx basic auth creds here, email can be blank
    Username: dockeradmin
    Email:                          # This we can leave blank
    Login Succeeded                                         # Successful login message

Once the login is succeeded, lets add some base docker images to our Private registry.

$ docker pull ubuntu:12.04                             #  Pulling the latest image of Ubuntu 12.04
    ubuntu:12.04: The image you are pulling has been verified
    ed52aaa56e98: Pull complete
    b875af6dcb23: Pull complete
    41959ee20b93: Pull complete
    f959d044ebdf: Pull complete
    511136ea3c5a: Already exists
    Status: Downloaded newer image for ubuntu:12.04

$ docker images                             # Check the downloaded images
    REPOSITORY                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
    ubuntu                      12.04               f959d044ebdf        5 days ago          130.1 MB

So, as per the Documentation, we need to create a tag for the image that we are going to push to our private registry. The tag must be of the syntax ”/:”

$ docker tag ubuntu:12.04     # We need to create a tag of format "<registry-server-fqdn>/<image-name>:<optional-tag>"

$ docker push                 # push the image to our new private registry
    The push refers to a repository [] (len: 1)
    Sending image list
    Pushing repository (1 tags)
    Image 511136ea3c5a already pushed, skipping
    ed52aaa56e98: Image successfully pushed
    b875af6dcb23: Image successfully pushed
    41959ee20b93: Image successfully pushed
    f959d044ebdf: Image successfully pushed
    Pushing tag for rev [f959d044ebdf] on {}

Let’s query the registry API for the pushed image

curl http://localhost:5000/v1/search
    {"num_results": 2, "query": "", "results": [{"description": "", "name": "library/wheezy"}, {"description": "", "name": "library/ubuntu"}]}

Currently both the Docker Client and Registry resides on the same machine, we can test push/pull image from a remote machine. The only dependency is we need to add the Self Signed CA to the trusted CA list, otherwise docker client will raise an SSL error while trying to login against the private registry.

Now let’s try pulling the images from the private registry.

$ docker pull
    Pulling repository
    f959d044ebdf: Download complete
    511136ea3c5a: Download complete
    ed52aaa56e98: Download complete
    b875af6dcb23: Download complete
    41959ee20b93: Download complete
    Status: Downloaded newer image for

$ docker images
    REPOSITORY                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
    debian                      wheezy              479215127fa7        5 days ago          84.99 MB   latest              479215127fa7        5 days ago          84.99 MB   12.04               f959d044ebdf        5 days ago          130.1 MB

Setting up S3 Backend for Docker Registry

Docker registry by default supports S3 backend for storing the images. But if we are using S3, it’s better to cache the image locally so that we don’t have to fetch S3 all the time. Redis really comes to the rescue. We can set up Redis Server as an LRU Cache and can define the settings in the config.yml of the registry or as an env variable.

$ apt-get install redis-server

Once Redis server is installed, we need to define the maxmemory to be allocated for the cache and maxmemory-policy which tells Redis how to clean the old cache when the maxmemory limit is reached. Add below settings to the redis.conf file

maxmemory 2000mb              # i'm allocating 2GB of cache size

maxmemory-policy volatile-lru     # removes the key with an expire set using an LRU algorithm

Now let’s define the env variables so that docker-registry can use them while starting up. Add the below variables to the /etc/default/docker-registry file.



Let’s start the Docker Registry in foreground and see if it’s starting with Redis Cache.

root@docker:~# docker-registry

   13/Jan/2015:19:07:42 +0000 INFO: Enabling storage cache on Redis
   13/Jan/2015:19:07:42 +0000 INFO: Redis host: localhost:6379 (db0)
   13/Jan/2015:19:07:42 +0000 INFO: Enabling lru cache on Redis
       13/Jan/2015:19:07:42 +0000 INFO: Redis lru host: localhost:6379 (db0)
       13/Jan/2015:19:07:42 +0000 INFO: Enabling storage cache on Redis

The above logs shows us that registry has started with Redis cache. Now we need to setup the S3 backend storage. By default for dev env, defaul backend is file storage. We need to change it to S3 in the config.yml

dev: &dev
    <<: *s3                            #by default this will be local, which is local file storage
    loglevel: _env:LOGLEVEL:debug
    debug: _env:DEBUG:true
    search_backend: _env:SEARCH_BACKEND:sqlalchemy

Now if we check the config.yml, in the S3 backend section, the mandatory variables are the ones mentioned below. The boto variables are needed only if we are using any non-Amazon S3-compliant object store.

AWS_REGION   => S3 region where the bucket is located
AWS_BUCKET   => S3 bucket name
STORAGE_PATH => the sub "folder" where image data will be stored
AWS_ENCRYPT  => if true, the container will be encrypted on the server-side by S3 and will be stored in an encrypted form while at rest in S3. Default value is `True`
AWS_SECURE   => true for HTTPS to S3
AWS_KEY      => S3 Access key
AWS_SECRET   => S3 secret key

We can define the above variables in the /etc/default/docker-registry file. And we need to restart the registry process to make the changes effective.

$ docker-registry         

    13/Jan/2015:23:40:39 +0000 INFO: Enabling storage cache on Redis
    13/Jan/2015:23:40:39 +0000 INFO: Redis host: localhost:6379 (db0)
    13/Jan/2015:23:40:39 +0000 INFO: Enabling lru cache on Redis
    13/Jan/2015:23:40:39 +0000 INFO: Redis lru host: localhost:6379 (db0)
    13/Jan/2015:23:40:39 +0000 INFO: Enabling storage cache on Redis
    13/Jan/2015:23:40:39 +0000 INFO: Redis config: {'path': '/registry1', 'host': 'localhost', 'password': None, 'db': 0, 'port': 6379}
    13/Jan/2015:23:40:39 +0000 DEBUG: Will return docker-registry.drivers.s3.Storage
    13/Jan/2015:23:40:39 +0000 DEBUG: Using access key provided by client.
    13/Jan/2015:23:40:39 +0000 DEBUG: Using secret key provided by client.
    13/Jan/2015:23:40:39 +0000 DEBUG: path=/
    13/Jan/2015:23:40:39 +0000 DEBUG: auth_path=/my-docker/
    13/Jan/2015:23:40:39 +0000 DEBUG: Method: HEAD
    13/Jan/2015:23:40:39 +0000 DEBUG: Path: /
    13/Jan/2015:23:40:39 +0000 DEBUG: Data:
    13/Jan/2015:23:40:39 +0000 DEBUG: Headers: {}
    13/Jan/2015:23:40:39 +0000 DEBUG: Host:
    13/Jan/2015:23:40:39 +0000 DEBUG: Port: 443
    13/Jan/2015:23:40:39 +0000 DEBUG: Params: {}
    13/Jan/2015:23:40:39 +0000 DEBUG: establishing HTTPS connection:, kwargs={'port': 443, 'timeout': 70}
    13/Jan/2015:23:40:39 +0000 DEBUG: Token: None
    13/Jan/2015:23:40:39 +0000 DEBUG: StringToSign:

    Tue, 13 Jan 2015 23:40:39 GMT
    13/Jan/2015:23:40:39 +0000 DEBUG: Signature:
    AWS XXXXXXXXXXXXXXXXXXXX:********************
    13/Jan/2015:23:40:39 +0000 DEBUG: Final headers: {'Date': 'Tue, 13 Jan 2015 23:40:39 GMT', 'Content-Length': '0', 'Authorization': u'AWS XXXXXXXXXXXXXXXXXXXX:********************', 'User-Agent': 'Boto/2.34.0 Python/2.7.3 Linux/3.8.0-44-generic'}
    13/Jan/2015:23:40:39 +0000 DEBUG: Response headers: [('x-amz-id-2', '*************************************************'), ('server', 'AmazonS3'), ('transfer-encoding', 'chunked'), ('x-amz-request-id', 'XXXXXXXXXXXXXX'), ('date', 'Tue, 13 Jan 2015 23:40:40 GMT'), ('content-type', 'application/xml')]
    13/Jan/2015:23:40:39 +0000 INFO: Boto based storage initialized
    2015-01-13 23:40:39 [21909] [INFO] Starting gunicorn 19.1.0
    2015-01-13 23:40:39 [21909] [INFO] Listening at: (21909)
    2015-01-13 23:40:39 [21909] [INFO] Using worker: gevent
    2015-01-13 23:40:39 [21919] [INFO] Booting worker with pid: 21919
    2015-01-13 23:40:39 [21920] [INFO] Booting worker with pid: 21920
    2015-01-13 23:40:39 [21921] [INFO] Booting worker with pid: 21921
    2015-01-13 23:40:39 [21922] [INFO] Booting worker with pid: 21922
    2015-01-13 23:40:39 [21909] [INFO] 4 workers

So now we have the Docker registry with S3 backend and Redis cache. Let’s push one of our local image see if registry can upload it to the S3 bucket.

$ docker push

    The push refers to a repository [] (len: 1)
    Sending image list
    Pushing repository (1 tags)
    511136ea3c5a: Image successfully pushed
    1aeada447715: Image successfully pushed
    479215127fa7: Image successfully pushed
    3192d5ea7137: Image successfully pushed
    Pushing tag for rev [3192d5ea7137] on {}

Now let’s check debug logs to have a more glimpse on what’s happening in the background.

# Sample output of S3 image upload

    Tue, 13 Jan 2015 20:05:05 GMT
    13/Jan/2015:20:05:05 +0000 DEBUG: Signature:
    AWS XXXXXXXXXXXXXXXXXXXX:********************
    13/Jan/2015:20:05:05 +0000 DEBUG: Final headers: {'Date': 'Tue, 13 Jan 2015 20:05:05 GMT', 'Content-Length': '0', 'Authorization': u'AWS   XXXXXXXXXXXXXXXXXXXX:********************', 'User-Agent': 'Boto/2.34.0 Python/2.7.3 Linux/3.8.0-44-generic'}
    13/Jan/2015:20:05:05 +0000 DEBUG: Response headers: [('x-amz-id-2', '*****************************************************'), ('server', 'AmazonS3'), ('transfer-encoding', 'chunked'), ('x-amz-request-id', 'XXXXXXXXXXXXX'), ('date', 'Tue, 13 Jan 2015 20:05:05 GMT'), ('content-type', 'application/xml')]
    13/Jan/2015:20:05:05 +0000 DEBUG: path=/registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/_inprogress
    13/Jan/2015:20:05:05 +0000 DEBUG: auth_path=/my-docker/registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/_inprogress
    13/Jan/2015:20:05:05 +0000 DEBUG: Method: PUT
    13/Jan/2015:20:05:05 +0000 DEBUG: Path: /registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/_inprogress
    13/Jan/2015:20:05:05 +0000 DEBUG: Data:
    13/Jan/2015:20:05:05 +0000 DEBUG: Headers: {'Content-MD5': u'xxxxxxxxxxxxxxx', 'Content-Length': '4', 'Expect': '100-Continue', 'x-amz-server-side-encryption': 'AES256', 'Content-Type': 'application/octet-stream', 'User-Agent': 'Boto/2.34.0 Python/2.7.3 Linux/3.8.0-44-generic'}
    13/Jan/2015:20:05:05 +0000 DEBUG: Host:
    13/Jan/2015:20:05:05 +0000 DEBUG: Port: 443
    13/Jan/2015:20:05:05 +0000 DEBUG: Params: {}
    13/Jan/2015:20:05:05 +0000 DEBUG: Token: None
    13/Jan/2015:20:05:05 +0000 DEBUG: StringToSign:
    Tue, 13 Jan 2015 20:05:05 GMT
    13/Jan/2015:20:05:05 +0000 DEBUG: Signature:
    AWS XXXXXXXXXXXXXXXXXXXX:********************
    13/Jan/2015:20:05:05 +0000 DEBUG: Final headers: {'Content-MD5': 'xxxxxxxxxxxxx', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue,              13 Jan 2015 20:05:05 GMT', 'x-amz-server-side-encryption': 'AES256', 'Content-Type': 'application/octet-stream', 'Authorization': u'AWS XXXXXXXXXXXXXXXXXXXX:********************', 'User-Agent': 'Boto/2.34.0 Python/2.7.3 Linux/3.8.0-44-generic'}
    13/Jan/2015:20:05:05 +0000 DEBUG: Response headers: [('content-length', '0'), ('x-amz-id-2', '*************************************'), ('server', 'AmazonS3'), ('x-amz-request-id', 'xxxxxxxxxxxxxx'), ('etag', '"b326b5062b2f0e69046810717534cb09"'), ('date', 'Tue, 13 Jan 2015 20:05:06 GMT'), ('x-amz-server-side-encryption', 'AES256')]
    13/Jan/2015:20:05:05 +0000 DEBUG: path=/registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/_checksum
    13/Jan/2015:20:05:05 +0000 DEBUG: auth_path=/my-docker/registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/_checksum
    13/Jan/2015:20:05:05 +0000 DEBUG: Method: HEAD
    13/Jan/2015:20:05:05 +0000 DEBUG: Path: /registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/_checksum
    13/Jan/2015:20:05:05 +0000 DEBUG: Data:
    13/Jan/2015:20:05:05 +0000 DEBUG: Headers: {}
    13/Jan/2015:20:05:05 +0000 DEBUG: Host:
    13/Jan/2015:20:05:05 +0000 DEBUG: Port: 443
    13/Jan/2015:20:05:05 +0000 DEBUG: Params: {}
    13/Jan/2015:20:05:05 +0000 DEBUG: Token: None
    13/Jan/2015:20:05:05 +0000 DEBUG: StringToSign:

    Tue, 13 Jan 2015 20:05:05 GMT
    13/Jan/2015:20:05:05 +0000 DEBUG: Signature:
    AWS XXXXXXXXXXXXXXXXXXXX:********************
    13/Jan/2015:20:05:05 +0000 DEBUG: Final headers: {'Date': 'Tue, 13 Jan 2015 20:05:05 GMT', 'Content-Length': '0', 'Authorization': u'AWS   XXXXXXXXXXXXXXXXXXXX:********************', 'User-Agent': 'Boto/2.34.0 Python/2.7.3 Linux/3.8.0-44-generic'}
    13/Jan/2015:20:05:05 +0000 DEBUG: Response headers: [('x-amz-id-2', '***************************************************'), ('server', 'AmazonS3'), ('transfer-encoding', 'chunked'), ('x-amz-request-id', 'xxxxxxxxxxxxxxx'), ('date', 'Tue, 13 Jan 2015 20:05:05 GMT'), ('content-type', 'application/xml')]
    13/Jan/2015:20:05:05 +0000 DEBUG: path=/
    13/Jan/2015:20:05:05 +0000 DEBUG: auth_path=/my-docker/
    13/Jan/2015:20:05:05 +0000 DEBUG: path=/?delimiter=/&prefix=registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/_checksum/
    13/Jan/2015:20:05:05 +0000 DEBUG: auth_path=/my-docker/? delimiter=/&prefix=registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/_checksum/
    13/Jan/2015:20:05:05 +0000 DEBUG: Method: GET
    13/Jan/2015:20:05:05 +0000 DEBUG: Path: /?delimiter=/&prefix=registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/_checksum/
    13/Jan/2015:20:05:05 +0000 DEBUG: Data:
    13/Jan/2015:20:05:05 +0000 DEBUG: Headers: {}
    13/Jan/2015:20:05:05 +0000 DEBUG: Host:
    13/Jan/2015:20:05:05 +0000 DEBUG: Port: 443
    13/Jan/2015:20:05:05 +0000 DEBUG: Params: {}
    13/Jan/2015:20:05:05 +0000 DEBUG: Token: None
    13/Jan/2015:20:05:05 +0000 DEBUG: StringToSign:
    13/Jan/2015:20:05:07 +0000 DEBUG: args = {'image_id': u'3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63'}
    13/Jan/2015:20:05:07 +0000 DEBUG: path=/registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/layer
    13/Jan/2015:20:05:07 +0000 DEBUG: auth_path=/my-docker/registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/layer
    13/Jan/2015:20:05:07 +0000 DEBUG: Method: HEAD
    13/Jan/2015:20:05:07 +0000 DEBUG: Path: /registry1/images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/layer
    13/Jan/2015:20:05:07 +0000 DEBUG: Data:
    13/Jan/2015:20:05:07 +0000 DEBUG: Headers: {}
    13/Jan/2015:20:05:07 +0000 DEBUG: Host:
    13/Jan/2015:20:05:07 +0000 DEBUG: Port: 443
    13/Jan/2015:20:05:07 +0000 DEBUG: Params: {}
    13/Jan/2015:20:05:07 +0000 DEBUG: Token: None
    13/Jan/2015:20:05:07 +0000 DEBUG: StringToSign:

    Tue, 13 Jan 2015 20:05:07 GMT
    13/Jan/2015:20:05:07 +0000 DEBUG: Signature:
    AWS XXXXXXXXXXXXXXXXXXXX:********************
    13/Jan/2015:20:05:07 +0000 DEBUG: Final headers: {'Date': 'Tue, 13 Jan 2015 20:05:07 GMT', 'Content-Length': '0', 'Authorization': u'AWS   XXXXXXXXXXXXXXXXXXXX:********************', 'User-Agent': 'Boto/2.34.0 Python/2.7.3 Linux/3.8.0-44-generic'}   
    13/Jan/2015:20:05:07 +0000 DEBUG: Token: None
    13/Jan/2015:20:05:07 +0000 DEBUG: StringToSign:

Now let’s query the registry for the newly uploaded image

$ curl -k


$ curl http://localhost:5000/v1/search
    {"num_results": 1, "query": "", "results": [{"description": "", "name": "mydocker/debian"}]}

Also let’s see if our Redis server is caching the image.

$ redis> keys *
 1) "cache_path:/tmp/registryrepositories/mydocker/debian/tag_wheezy"
 2) "cache_path:/registry1images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/ancestry"
 3) "cache_path:/registry1images/1aeada4477158496dc31ee5c6e7174240140d83fddf94bc57fc02bee1b04e44f/json"
 4) "cache_path:/registryimages/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/ancestry"
 5) "diff-worker"
 6) "cache_path:/registry1images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json"
 7) "cache_path:/registry1repositories/mydocker/debian/_index_images"
 8) "cache_path:/registryimages/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/_inprogress"
 9) "cache_path:/registry1images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/_checksum"
10) "cache_path:/registry1repositories/mydocker/debian/tagwheezy_json"
11) "cache_path:/registryimages/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/_checksum"
12) "cache_path:/registry1images/479215127fa7b852902ed734f3a7ac69177c0d4d9446ad3a1648938230c3c8ab/ancestry"
13) "cache_path:/registry1repositories/mydocker/debian/tag_wheezy"
14) "cache_path:/registry1images/1aeada4477158496dc31ee5c6e7174240140d83fddf94bc57fc02bee1b04e44f/ancestry"
15) "cache_path:/registry1images/3192d5ea7137e4f47f4624a5cc7786af2159a44f49511aeed28aa672416cec63/json"
16) "cache_path:/registryrepositories/mydocker/debian/_index_images"
17) "cache_path:/registryimages/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json"
18) "cache_path:/registry1images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/ancestry"
19) "cache_path:/registry1images/479215127fa7b852902ed734f3a7ac69177c0d4d9446ad3a1648938230c3c8ab/_checksum"
20) "cache_path:/registry1images/1aeada4477158496dc31ee5c6e7174240140d83fddf94bc57fc02bee1b04e44f/_checksum"
21) "cache_path:/registry1images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/_checksum"
22) "cache_path:/registry1images/479215127fa7b852902ed734f3a7ac69177c0d4d9446ad3a1648938230c3c8ab/json"

Now for those who want to have a Continous Integration system, we can set up Jenkins to build the autmated images and upload to our Private registry and use Mesos/CoreOS to deploy the image through out our infrastructure in a fully automated fashion.