Automating Debian Package Management

With the rise of CI tools like Jenkins/Gitlab and Config management tools like Salt/Ansible, Continous integration became so flexible. Now if we check, most of the Projects are using GIT as a Version control and CI tools like Jenkins to build and test the packages automatically whenever any change is pushed to the repo. And finally once the build is successful, the packages are pushed to repo so that config management systems like Salt/Puppet/Ansible can go ahead and perform the upgrade. In my previous blogs, i’ve explained on how to build a Debian package and how to create and manage APT repo’s via aptly. In this blog i’ll explain how to automate these two processes.

So the flow is like this. We have a Github repo, and once a changed is pushed to the repo, Github will send a hook to our Jenkins server which in turn triggers the Jenkins package build. Once the package has been successfully built, jenkins will automatically add the new packages to our repo and publish the same to our APT repo via aptly

Installing Jenkins

First, let’s setup a Jenkins build server.

$ wget -q -O - https://jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -

$ echo "deb http://pkg.jenkins-ci.org/debian binary/" > /etc/apt/sources.list.d/jenkins.list

$ apt-get update && apt-get install jenkins

$ /etc/init.d/jenkins restart

Once the Jenkins service is started, we can access the Jenkins UI via ”http://jenkins-server-ip:8080”. By default there is no authentication for this URL, so accessing the URL will open up the Jenkins UI.

Creating a Build Job in Jenkins

In order to use a Git repo, we have to install the Git plugin first. In Jenkins UI, Go to ”Manage Jenkins” – > ”Manage Plugins” – > ”Available” and search for ”GIT plugin” and install it. Once the Git plugin has been installed we can create a new build job.

Click on ”New Item” on the Home Page and Select ”Freestyle Project” and Click on “OK”. On the Next page, we need to configure all the necessary steps for build job. Fill in the necessary details like Project Name, Description etc. Under “Source Code Management”, select Git and enter the Repo URL. Make sure that the jenkins user has access to the repo. We can also use Deploy keys, but i’ve generated a separate ssh key for Jenkins user and the same has been added to Github. Under ”Build Triggers” select ‘Build when a change is pushed to GitHub’ so that Jenkins will start the build job everytime when a change has been pushed to repo.

Under the Build section, Click on ”Add build step” and select ’Execute shell’ and let’s add our package build script which is stage 1.

set -e
set -x
debuild -us -uc

In Stage 2, i’m going publish my newly built packages to my APT repo

aptly repo add myapt ../openvpn*.deb
/usr/bin/env script -qfc "aptly publish -passphrase=<GPG passphrase> update myapt"

If you see my above command, i’ve used the script command. This is because, i was getting this error “aptly stderr: gpg: cannot open tty /dev/tty': No such device or address“, whenever i try to update a repo via aptly using Jenkins. This is due to a bug in aptly. The fix has been placed on the Master branch but its not yet released. The script command is a temporary work around for this bug.

Now we have a Build job ready. We can manually trigger a build to test if the Job is working fine. If the build is successfull, we are done with our build server. Now the final step is Configuring Github to send a trigger whenever any change is pushed to Github.

Configuring Github Triggers

Go the Github repo and Click on the Repo settings. Open ”Webhooks and Services” and select ”Add Service” and select ”GitHub plugin“.Now it will ask for Jenkin’s Hook URL, which is ”http://:8080/github-webhook/” and add the service. Once the service is set, we can click on “Test service” to check if the webhook is working fine.

Once the test hook is created, go to the Jenkins job page and select ”GitHub Hook Log”. The test hook should get displayed there. If not there is something wrong on the config.

Now we have a fully automated build and release management. Config management tools like Salt/Ansible etc.. can go ahead and start the deployment process.

Managing Debian APT Repository via Aptly

In my previous blog, i’ve explained how to build a Debian pacakge from source. In this blog i’m to explain how to create and manage our own apt repository. Enter aptly,is a swiss army knife for Debian repository management: it allows us to mirror remote repositories, manage local package repositories, take snapshots, pull new versions of packages along with dependencies, publish as Debian repository. Aptly can upload the repo to Amazon S3, but we need to install APT S3 support, in order to use it from S3.

First, let’s install aptly on our build server. A more detailed documentation on installation is available in the website

$ echo "deb http://repo.aptly.info/ squeeze main" > /etc/apt/sources.list

$ gpg --keyserver keys.gnupg.net --recv-keys 2A194991

$ gpg -a --export 2A194991 | sudo apt-key add -

$ apt-get update && apt-get install aptly

Let’s create a repo,

$ aptly repo create -distribution=wheezy -component=main my-repo    # where my-repo is the name of the repository

Once the repo is created, we can start adding our newly created packages to our new repo.

$ aptly repo add <repo name> <your debian file>    # in my case aptly repo add myrepo openvpn_2.3.6_amd64.deb

The above command will add the new package to the repo. Now in order to make this repo usable, we need to publish this repo. A valid GPG key is required for publishing the repo. So let’s create the gpg key for aptly.

$ gpg --gen-key

$ gpg --export --armor <email-id-used-fo-gpg-key-creation> > myrepo-pubkey.asc   # creates a pubkey that distributed

$ gpg --send-key KEYNAME     # This command can be used if we want to send the key to a public server, we can also pass --keyserver <server-url>, if we want to specifiy a specific keyserver

Once we have our GPG key, we can publish our repo. By default aptly can publish the repo to S3 or it can publish it locally and we can use any webserver to servce this repo.

$ aptly publish --distribution="wheezy" repo my-repo

Once published, we can point the webserver to “~/.aptly/”, where our repo files will be created. Aptly also comes with an embedded webserver which can be invoked by running aptly serve. Aptly really makes the repo management so easy. We can actually integrate this into our jenkins job so that each time when we build a package, we can directly add and upload the same to our repository.

Building a Debian Package

Installing applications via packages saves us a lot of time. Especially being an OPS oriented guy, compiling applications from source is somtimes pain and time consuming. Especially the dependencies. But later, after the rise of config management system, people started creating corresponding automated scripts that will install necessary dependencies and the ususal make && make install. But if you check applications like Freeswitch was taking 15+min to finish compiliations, which is defintely a bad idea when you want to deploy the a new patch on a cluster. In such cases packages are really a life saver. Build the packages once as per our requirement and deploy it throughout the infrastructure. Now with the tools like jenkins,TravisCI etc we can attain a good level of CI.

In this blog, i’m going to explain on how to build a debian package from scratch. First let’s install two main dependencies for a build machine

$ apt-get install devscripts build-essential

For the past few days i was playing with OpenVPN and SoftEther. I’m going to build a simple debian package for OpenVPN from source. The current stable version of OpenVPN is available 2.3.6. First let’s get the OpenVPN source code.

$ wget http://swupdate.openvpn.org/community/releases/openvpn-2.3.6.tar.gz

$ tar xvzf openvpn-2.3.6.tar.gz && cd openvpn-2.3.6

Now for building a package, first we need to create a debian folder. And in this folder we are going to place all necessary files required for building a package.

$ mkdir debian

As per the Debian pacakging Documentation, the mandatory files are rules control, changelog. Changlog file content should match the exact syntax, otherwise packaging will fail at the initial stage itself. There are some more optional files that we can use. Below are the files present in my debian folder

changelog           => Changelog for my Package
control             => Contains Details about the package including the dependencies
dirs                => specifies any directories which we need but which are not created by the normal installation procedure, handled by 'dh_installdirs'
openvpn.default     => this file will be copied to /etc/default/openvpn
openvpn.init        => this file will be copied to /etc/init.d/openvpn, handled by 'dh_installinit'
postinst.debhelper  => Any action that need to be performed once the package installation is completed, like creating a specific user, starting service etc
postrm.debhelper        => Any action that need to be performed once the package removal is completed, like deleting a specific user
prerm.debhelper     => Any action that need to be performed before the package removal is initiated, like stopping the service
rules           => Contains rules for build procedure

In my case i wanted to install the openvpn on a custom location say ’/opt/openvpn’. So if we are building from scratch manually, we can mention the prefix like ’./configure –prefix=/opt/openvpn’. but in the build process, dh_auto_configure is running our ’./configure’ operation with dfault option ie, no custom prefix. So we need to overide this process if we want to have a custom prefix. Below is the content of my rules file.

# rules file

    #!/usr/bin/make -f
    # vim: tabstop=4 softtabstop=4 noexpandtab fileencoding=utf-8

    # Uncomment this to turn on verbose mode.
    export DH_VERBOSE=1

    DEB_DIR=$(CURDIR)/debian/openvpn

    %:
        dh $@
    override_dh_auto_configure:                      # override of configure
        ./configure --prefix=/opt/openvpn

Once we have all the necessary files in place, we can start the build process. Make sure that all the dependency packages mentioned in the control file is installed on the build server.

    $ debuild -us -uc

If the build command is completed successfully, we will see the deb package as well as the source package just above our openvpn source folderm which is the default path where dh_builddeb places the files. We can overide the same too.

#!/usr/bin/make -f
# vim: tabstop=4 softtabstop=4 noexpandtab fileencoding=utf-8

# Uncomment this to turn on verbose mode.
export DH_VERBOSE=1

DEB_DIR=$(CURDIR)/debian/openvpn

%:
    dh $@
override_dh_auto_configure:
    ./configure --prefix=/opt/openvpn
override_dh_builddeb:
    dh_builddeb --destdir=./deb-pkg/

So now we have the Debian package. We can test installing it manually via ’dpkg -i’. This was just a go thorugh on how to build a simple debian package. In my next blog, i’ll be discussing about how to create and manage a private apt repository using a awsme tool called aptly