5 tips for flexible and scalable CI with Docker

As Docker mentions in their use case, using it for Continuous Integration and Continuous Delivery (CI/CD) is an excellent way to utilize the tool outside the production system. At Nulab we have been using Docker in this way since we began offering pull requests for Backlog. Here, I will introduce five tips we’ve learned in the process.

Overview of Nulab’s CI Environment

This graphic presents an overview of Nulab’s CI Environment:

Jenkins is the center of our CI environment. Backlog and Typetalk trigger jobs in Jenkins. In actual setting, we use Jenkins Backlog Plugin and Jenkins Typetalk Plugin to work those tools together. I’ve added a reference at the end of this post, for those who are interested.

Job execution is done on the slave side, with EC2 mainly for testing and an exclusive slave for special environments, like mobile builds. Currently, we have over 10 servers in the cluster, including the master, and average 250 builds a day.

Let me now introduce you to some more detailed points I take care of when operating this CI environment.

Five Tips for Docker in CI

1. Keep Slave Configuration Method Simple

The first tip is to keep slave configuration method simple. We set up a slave using Jenkins AWS EC2 Plugin with the following simple rules:

  • Use the latest version of Amazon Linux for AMI as it is
  • Simply install Docker and Docker compose when starting EC2 instance

This shortens the slave’s startup time and also creates a CI environment that will work anywhere so long as Docker is running.

2. Test in a Single Dockerfile

It is not rare to run a test that requires database or other middlewares. In general, it’s better to have one process per Docker container. For testing, though, we consider it OK to have several processes running in one container.

In the example below, we install Redis in a Docker container, and start it and the Java processes for testing.

Dockerfile example

FROM java:openjdk-8
# install redis
RUN apt-get install –y redis-server

Test job example

docker run ${TEST_IMAGE} bach –c "service redis-server start ; ./gradlew clean test"

This enables us to run test easily without changing test settings like database server name and port from a local development environment.

Some of you may think of Docker Compose as another approach. In fact, we also use it for some parts of our project. But, aside from cases where you would want to use the Dockerfile for something other than testing, the method described above is usually good enough.

3. Use Cache Effectively

To keep build time short, it’s important to use the cache effectively. Let’s look at how to do this from two points of view: Docker image and dependent library.

Use Docker Image of In-House Registry

We have an in-house registry of our own (see below) and save custom images there. We create custom images by adding required runtimes like JDK, Perl, Python etc to publicly available images.

You should place in-house registry near slaves so that custom images will be downloaded quickly. In our case, we have the in-house registry in AWS Tokyo region, since that’s where our slaves are running. This shortens the build time, compared to using a public image, because it can be downloaded faster and contains everything necessary for the build.

Cache Dependent Libraries

Cacheing the dependency library is the most important point for building a Java project. We do it in the following three patterns.

Host Directory Pattern

The first pattern uses a host directory as a cache. When running Docker container, mount host directory to directory inside the container where dependent libraries are stored (see below).

docker run –v ${HOME}/.gradle:/root/.gradle ${TEST_IMAGE} ./gradlew clean test

If you don’t have dependent libraries in host directory, they will be resolved and downloaded on the first run. You can then use the resultant dependent libraries from host directory as your cache. In case of Java, there’re commonly used libraries across projects and that increases cache effectiveness. A drawback, however, is the possibility of permission issues for the host directory. I will address it later in the section about deleting the container after the build.

Cache in Advance Pattern

The second pattern is to build the Docker image with dependent libraries in advance of testing and use the image as a cache.

First, create the Dockerfile:

RUN mkdir -p /opt/app
COPY requirements.txt /opt/app/
WORKDIR /opt/app
RUN pip install -r requirements.txt

COPY . /opt/app

Next, run it:

docker build –t ${TEST_IMAGE} .
docker run ${TEST_IMAGE} py.test tests

This method works best if the dependencies (specified in requirements.txt, above) do not change often. In the example above, we install dependent libraries by “RUN pip install -r requirements.txt” and they will be cached in Docker until requirements.txt will change. Since everything is included in the Docker container, we avoid the permission problem previously mentioned, but this method has its own drawback: if you change a single dependency, the entire cache is cleared.

External Cache Pattern

Finally, we have the external cache pattern. Here we took a hint from Travis CI’s approach.In this method, you make Dockerfile like this:

RUN mkdir /root/.gradle
RUN cd /root/.gradle; curl -skL https://s3-ap-northeast-1.amazonaws.com/${CACHE_BUCKET}/cache.20151201.tar.gz | tar zxf -

This very simple approach is much faster than making Gradle or sbt resolve and download the dependencies, even for the first build activation right after starting the slave. But the drawback is that you need to maintain an external cache.

Each approach has its own advantages and drawbacks. In many cases, we can run a test of Java project as root user and thus adopt the first approach without drawback. For Phython and Perl projects, however, it is much easier to debug if the dependencies are in the container, so here we use the second approach. The third approach is especially useful to Java projects, so we are currently considering if we can adopt this method mixed with the host directory method.

4. Remove Container After Build

If you don’t remove unnecessary containers after running the test, they will eventually use up host storage. Before removing the container, however, you must save the build result, result report, and the .war file. Otherwise, everything will be gone.

Here are two ways to remove containers after you get the build result:

Mount Jenkins’ Workspace

The first method is to mount Jenkins’ workspace to working directory of container. In the following example, first we set WORKDIR to/opt/app and then run docker.

docker run --rm –v $(pwd):/opt/app ${TEST_IMAGE} ./gradlew clean test

As normal build tool often stores its result under working directory, you don’t have to explicitly retrieving the build result from container. The result will just remain in Jenkins’ workspace after the test.

However, if test needs to be run by non root user,  you have to consider permission of host directory within the Docker container. We met this issue in installing several libraries by npm, or using library like testing.postgresql. If you don’t care about it, some problems may occur, such as failing to write the build result or to build itself at the next run.

We solve this permission problem in either of the following two ways.

In the first method, we write the result in directory where the execution user has permission to write when running build. Then we copy that file after build. This is illustrated below, in a file called run.sh:

su test-user –c "py.test tests –-junit-xml=/var/tmp/results.xml"
cp –p /var/tmp/results.xml .

Then we run docker as below:

docker run --rm –v $(pwd):/opt/app ${TEST_IMAGE} ./run.sh

In the second method, we change the owner of the directory to the execution user at the beginning of build, and change it back after build is done. This is illustrated below, as a different run.sh file:

chown test-user .
su test-user -c "py.test tests"
chown $1 .

Then we run docker as below:

docker run --rm –v $(pwd):/opt/app ${TEST_IMAGE} ./run.sh $(id -u)

Here we pass uid in host as an argument to the script. There may be other approaches such as getting ownership of directory inside container.

Get Result After Test Run, then Remove the Container

In the other approach to properly removing containers, we get the results file from the container after build, like this:

UNIQUE_NAME=“TEST_${GIT_COMMIT}_$(date +%s)”
docker run --name=${UNIQUE_NAME} ${TEST_IMAGE} ./gradlew clean test
docker cp ${UNIQUE_NAME}:/opt/app/build/test-result/ test-result
docker rm ${UNIQUE_NAME}

n this approach you won’t encounter the permissions problem mentioned earlier, but there are a few steps involved in removing the container etc.

Both methods have advantages and disadvantages, but so far we’ve adopted the first approach, even though it is a little complicated, because it is less likely to leave trash.

5. Dockernize Tools Required for Job Execution

We created our own tool to upload build archive (mainly .war and zip file of Play! Framework, etc.) and static resources inside of the archive to S3. Originally we used to install the tool to the slave and run it in job settings like this:

/usr/local/bin/upload-static-s3 ROOT.war -b ${S3_CDN_BUCKET}

That is what we used to do. Now, we create the custom image including the tool and set ENTRYPOINT in the image like this.

ENTRYPOINT ["/usr/local/bin/upload-static-s3"]

The image is stored in our in-house registry and we run docker as below, using as same arguments as the ones we used before

docker run --rm ${IN_HOUSE_REGISTORY_URL}/upload-static-s3 ROOT.war –b ${S3_CDN_BUCKET}

We wrote this tool in Go language for easy installation to the slave, but with Dockernize that became unnecessary. Finally we can run the whole CI process of ‘test, build, upload to s3′ by Docker alone.

Advantages of Using Docker in CI

After adopting Docker, we found we gained three advantages:

  • We can now run test for all pull request branches
  • We have improved build performance
  • We can now run CI wherever Docker is running

Regarding the first point, slave setup became easier and now we can run it in an independent environment, so we can automatically run test for each pull request.

Regarding the second point, slave changes have become easier, so when performance is not good we can easily solve the problem by using a higher spec instance or adding more slaves. As a result we managed to almost halve the build time after introducing Docker and instance type changes, as shown below in a graph of the build times for one project.

Regarding the third point, now we can run the whole CI process with Docker, so we can run it not only in AWS but also in other cloud environments. CI/CD environment is now indispensable for service operation. We think the ability to reconstruct easily in a different environment is important for service availability.

A disadvantage, on the other hand, is that not all application developers are familiar with Docker, yet familiarity with Docker becomes essential for maintaining each job, so it makes maintenance of CI job harder for some members in our team. But Docker’s useful points are significant, and spread beyond CI, so we feel it is warranted to introduce it to our development process, and educate our team so they become familiar with it.


These days, I’ve had more opportunities to see and hear about examples of how Docker is used in production. However, it’s not easy to adopt what I learned to our specific environment, because there are so many unique specifics, conditions, and features for every project. In this regard, the CI environment seems the most uniform, and thus the easiest to place to begin with Docker – it is easier to learn from and adopt practices from other examples. Considering this along with the clear advantages I mentioned in this post, CI/CD seems the best place to try Docker in our work flow, allowing our team to gain experience with it.

We are hiring in New York, Tokyo, Kyoto and Fukuoka now!  For more details

References

Try Backlog for 30 days.

Join 800,000 developers running on Backlog. No credit card required.

Try It Free