TIL – Dockerfiles are really sensitive to OOO

The Land of Ooo

TIL – Order of operations are important!

Today I learned that order can have a profound effect on the size of the resulting images. For example my first Dockerfile contained a section something like this:

...
RUN curl http://www.bigfiles.com/bigfile.tgz > /opt/bigfile.tgz && \
    tar -C /opt -xzf /opt/bigfile.tgz
...

Which produced a ~300 MiB image.
But wait… the permissions are wrong! Let’s fix that:

...
RUN curl http://www.bigfiles.com/bigfile.tgz > /opt/bigfile.tgz && \
    tar -C /opt -xzf /opt/bigfile.tgz
RUN chown -R user:user /opt/bigfiledir
...

Wow! Suddenly we are over 500MiB! Why?

AuFS is actually storing the files twice. Resulting in an extra 200MiB just to store permissions. Yikes!

So what’s the solution?

...
RUN curl http://www.bigfiles.com/bigfile.tgz > /opt/bigfile.tgz && \
    tar -C /opt -xzf /opt/bigfile.tgz && \
    chown -R user:user /opt/bigfiledir
...

Since this all happens in one command the changes don’t get saved in an overlay layer.

My actual work was much more complicated so it took awhile to spot this, but it’s an important lesson nonetheless.

Keep your images small folks!

CoreOS + Etcd + Flannel = Pretty Cool

This is just a brief update on what I’m looking into.

I have a cluster, how do containers on different machines talk?

An easy solution is Flannel + Etcd. Here’s a simple example which will work on two machines. (I suggest using coreos-vagrant to create a two node cluster with flannel. Here’s my user-data and config.rb.)

First Machine:

CID=$(docker run -d alpine /bin/sh -c "while true; do sleep 1; done") && etcdctl set /server $(docker inspect --format '{{.NetworkSettings.IPAddress}}' $CID) && docker exec -it $CID /usr/bin/nc -l -p 9999 && docker stop --time=0 $CID && docker rm $CID

Note: Exit with Ctrl+P Control+Q (^P^Q), doing a Control+C (^C) will not stop and remove the docker container.

Second Machine

docker run --rm -it --add-host server:$(etcdctl get /server) alpine /usr/bin/nc server 9999

Now you should be able to chat between the two machines!

Meh? What back magic is this?

I’m simply using etcd to store the flannel IP of the server in the value of the /server key. On another machine we use that IP to attach netcat. The black magic is in flannel’s overlay network and etcd’s distributed key/value store.

Breaking apart the commands to the server:

CID=$(docker run -d --name server alpine /bin/sh -c "while true; do sleep 1; done")

This command sets up a docker container running the shell with a script that just sleeps. Simply running netcat doesn’t seem to give a good two-way connection in this example. The $CID part stores the docker container’s ID in a variable so I can use it later. I could have used the –name option to give a sensible name, but docker inspect doesn’t seem to accept a name.

etcdctl set /server $(docker inspect --format '{{.NetworkSettings.IPAddress}}' $CID)

Here I am setting a key in etcd so that another machine in fleet (which could be on the other side of the world) can access the flannel IP address of this server. This command makes use of the $CID variable set before (since it is random) and docker inspect’s format command to remove the formatting.

docker exec -it $CID /usr/bin/nc -l -p 9999

Executing netcat here instead of in the docker run phase is just for demonstration, normally this part would be done there. Because we are people who like to see things work, executing this command allows us to interact with the netcat server.

docker stop --time=0 $CID

Stopping the container with a zero time is analogous to killing it.

docker rm $CID

Get rid of that hanging container!

Lesson learned:

  1. Flannel works nicely
  2. Etcd is great
  3. Netcat is amazing
  4. Alpine is a wonderful ~5MiB distro for docker containers
  5. This stuff really needs a web UI! (If you know of a good one, leave a comment and I’ll blog about it.)