Docker is an awesome tool to build, manage and operate Linux containers. Docker has been gaining momentum for various use cases, and if you want to learn more I recommend checking out the Docker.io website.

I have been using Docker on and off for months, but recently started to deploy services using it onto the Rackspace Public Cloud. Unfortunately, not everything went smoothly, and this is the story of getting it all working.

tl;dr: Updated Ubuntu Images.

I have published updated Ubuntu 12.04 LTS images for Docker under the racker/precise-with-updates repository. Get the images by running docker pull racker/precise-with-updates. The latest tag is automatically updated every day with the latest Ubuntu updates and security patches. The Dockerfile for building this is on racker: docker-ubuntu-with-updates if you want to tweak it for your own uses.

Illegal instruction!?

Following the basic Docker instructions to get it running on Ubuntu 12.04, I installed a new kernel, rebooted, and dockerd was running successfully.

I started building a new Dockerfile from the ubuntu:12.04 base image. Everything was going OK, but then apt-get upgrade crashed with:

Illegal instruction

After some prodding, I find Docker issue #1984 — at least I’m not alone in my sorrow.

Digging through the links, you come to LP: eglibc/+bug/956051, a wonderful bug about glibc. glibc made incorrect the assumptions that if the FMA4 instruction set is available, that the Advanced Vector Extensions (AVX) would also be available. In LP: eglibc/+bug/979003 a patch for this bug is pushed to all recent Ubuntu versions, so why did it crash with Docker?

This affects Docker on Rackspace for two reasons:

1. Docker’s base Ubuntu 12.04 image has not applied any patches.
2. Many of Rackspace’s Public Cloud servers run an AMD Opteron(tm) 4332 processor with Xen which has FMA4 instructions but the AVX instructions do not work.

Since the bug has already been fixed upstream, creating an updated Ubuntu image to use as a base seems like the easiest way to fix this. Little did I know that creating an updated image was another rabbit hole.

Building an updated Ubuntu image.

I naively started by trying to just run apt-get dist-upgrade from the ubuntu:12.04 base image. It didn’t work at all. Packages tried to mess with upstart, and something went horribly wrong with /dev/shm.

After taking a break to propose to my fiancé and vacation in Hawaii, I built up the following Dockerfile:

FROM ubuntu:12.04

I started with the ubuntu:12.04 base image; It is possible to start from scratch using debootstrap, but it takes much longer to build and doesn’t provide any advantages.

To allow automated installation of new packages, we set the debconf(7) frontend to noninteractive, which never prompts the user for choices on installation/configuration of packages:

ENV DEBIAN_FRONTEND noninteractive

One of the updated packages is initramfs-tools. In the post-install hooks, it tries to update the initramfs for the machine and even try to run grub or lilo. Since we are inside docker we don’t want to do this. Debian bug #594189 contains many more details about these issues, but by setting the INITRD environment variable we can skip these steps:

ENV INITRD No

ischroot is a command used by many post install scripts in Debian to determine how to treat the machine. Unfortunately as discussed in in Debian bug #685034 it tends to not work correctly. Inside a Docker container we almost always want ischroot to return true. Because of this, I made a new ischroot which if the FAKE_CHROOT environment variable is set, always exits 0:

ENV FAKE_CHROOT 1
RUN mv /usr/bin/ischroot /usr/bin/ischroot.original
ADD src/ischroot /usr/bin/ischroot

Using this replacement ischroot, it allows updates to the initscripts package to successfully install without breaking /dev/shm, which works around LP Bug #974584.

policy-rc.d provides a method of controlling of the init scripts of all packages. By exiting with a 101, the init scripts for a package will not be ran, since 101 stands for action forbidden by policy:

ADD src/policy-rc.d /usr/sbin/policy-rc.d

Before installing any packages, I added a new sources.list that has -updates and -security repositories enabled and to use mirror.rackspace.com. Rackspace maintains mirror.rackspace.com as a GeoDNS address to pull from the nearest Rackspace data center, and it is generally much faster than archive.ubuntu.com.

ADD src/sources.list /etc/apt/sources.list

Normally dpkg will call fsync after every package is installed, but when building an image we don’t need to worry about individual fsyncs, so we can use the force-unsafe-io option:

RUN echo 'force-unsafe-io' | tee /etc/dpkg/dpkg.cfg.d/02apt-speedup

Since we want make the image as small as possible, we add a Post-Invoke hook to dpkg which deletes cached deb files after installation:

RUN echo 'DPkg::Post-Invoke {"/bin/rm -f /var/cache/apt/archives/*.deb || true";};' | tee /etc/apt/apt.conf.d/no-cache

Finally we run dist-upgrade:

RUN apt-get update -y && apt-get dist-upgrade -y

After all the upgrades have been applied, we want to clean out a few more cached files:

RUN apt-get clean
RUN rm -rf /var/cache/apt/* && rm -rf /var/lib/apt/lists/mirror.rackspace.com*

These final steps bring the image down to under 90 megabytes, which is smaller than the default ubuntu:12.04 image.

As one last trick, I flatten the image into a single layer by export and importing the image.

Automatically updated every day.

Ubuntu’s gets many updates and security patches on a regular basis. Rather than building a one off image, I hooked up Docker to a builder in Jenkins. The latest tag is rebuilt every day and pushed to the public registry under racker/precise-with-updates.

This means you can use racker/precise-with-updates:latest in a Dockerfile, or in your interactive terminals:

docker run -i -t racker/precise-with-updates /bin/bash

Using docker pull you can bring the images down locally for other operations:

docker pull racker/precise-with-updates
docker images racker/precise-with-updates

I have also published the source for the image on Github, and would welcome any PRs or feedback.