How Docker in Docker solved our concurrency problem in a second

A few weeks ago we encountered a problem where some of our customers were unable to run a job which was using a private repository for storing an image of their application.

The custom built service, Docker Runner, we’re using to provision Docker commands, ran into an issue where two jobs were trying to login to a registry, pull an image, and logout.

The problem occurred when two jobs were trying to pull an image from a repository using different accounts at the same time. First job’s command logged in (as a user A), then the second one (as a user B) and suddenly first job ended with an error (that user A is unable to find and pull the image). It was because user A was no longer logged in.

As this service is built continuously, adding more and more features, support for different registries, etc. we didn’t notice this problem at first. Bugs happen, unfortunately.

A proper fix would require executing a set of commands (docker login && docker pull && docker logout) in a different user’s shell — to ensure that every set of commands has its own separate environment.

But applying the above would probably will require more time and changes to how we execute docker commands on our workers.

Docker in Docker

Then we came up with an idea. Can we isolate the aforementioned set of commands to not to run in a different shell, but to run in a separate container? Let’s see.

Running Docker in Docker is perfect solution when it comes to isolation. Basically you’ll have a whole new machine.

So we changed the pulling code from:

docker login \
  && docker pull some-image \
  && docker logout

To:

docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  docker:1.11 \
  sh -c 'docker login && docker pull some-image && docker logout'

So pulling is done in a separate environment and will no longer conflict. And by mounting the host’s Docker socket (it’s still a file) all images pulled in the container will be available to run on the host.

It was a good choice, wasn’t it? :)