One of our highest priorities at Mist.io is to never break production. Our users depend on it to manage and monitor their servers and we depend on them. At the same time, we need to move fast with development and deliver updates as soon as possible. We want to be able to easily deploy several times per day.

A big part of Mist.io is the web interface so we use tools like Selenium, Splinter and Behave for headless web interface testing. However testing the UI is time-consuming. Having to wait 40 minutes to get a green light before merging each pull request is not very agile.

In this post we’ll describe the setup we’ve used to reduce testing time. It is based on Jenkins, Ansible and Docker and its main mission is to automate build steps and run tests in parallel and as quickly as possible.

Since execution time is essential for us, we opted to run our CI suite on one of Rackspace’s High-Performance Cloud Servers due to their fast performance and short provisioning times. In general, make sure you give your test server enough RAM and a fast disk.

Our first stop for our testing suite, is Jenkins. We’ve used Jenkins in other projects before and we feel comfortable with it since it is mature, widely used and provides great flexibility through its plugin system. Jenkins has very good Github integration which is another plus for us. It is quite simple to set it up so that every commit in a branch will trigger the tests.

When our needs started growing, our first thought was to add more Jenkins nodes and have multiple tests running in parallel. But there are many different environments that we want to test every commit against:

Test the deployment of the app in a clean environment, a fresh build.

Test the deployment of the app in a staging environment where you want to ensure backwards compatibility

Test the web interface against all supported browsers.

All these meant that testing requirements would grow over time and a testing infrastructure based solely on Jenkins would not do the job fast enough.

Enter Docker

Docker helps you easily create lightweight, portable, self-sufficient containers from any application. It is fast, reliable and a perfect fit for our needs. The idea, is to set up Jenkins so that every pull request to a specific branch (e.g. staging) triggers a job. Jenkins then commands our testing infrastructure to spawn different docker applications simultaneously and runs the tests in parallel.

This way, we can test any environment. We just have to describe each application the same way we would’ve done manually. On the plus side, we can use pre-made images from the docker repository to jumpstart our tests.

For example we could do this:

docker pull ubuntu

And we will end up with an ubuntu image/application. We can then run the docker container by typing:

docker run -i -t ubuntu /bin/bash

And we will be in a fresh, newly created ubuntu environment.

But we don’t want to manually run docker commands after every pull request. What we need, are Dockerfiles. Dockerfiles are sets of steps that describe an image/container. We’ll use them to build our custom docker images.

Dockerfile commands are fairly simple. For example:

FROM ubuntu:latest MAINTAINER mist.io RUN echo