TIL – Dockerfiles are really sensitive to OOO

The Land of Ooo

TIL – Order of operations are important!

Today I learned that order can have a profound effect on the size of the resulting images. For example my first Dockerfile contained a section something like this:

RUN curl http://www.bigfiles.com/bigfile.tgz > /opt/bigfile.tgz && \
    tar -C /opt -xzf /opt/bigfile.tgz

Which produced a ~300 MiB image.
But wait… the permissions are wrong! Let’s fix that:

RUN curl http://www.bigfiles.com/bigfile.tgz > /opt/bigfile.tgz && \
    tar -C /opt -xzf /opt/bigfile.tgz
RUN chown -R user:user /opt/bigfiledir

Wow! Suddenly we are over 500MiB! Why?

AuFS is actually storing the files twice. Resulting in an extra 200MiB just to store permissions. Yikes!

So what’s the solution?

RUN curl http://www.bigfiles.com/bigfile.tgz > /opt/bigfile.tgz && \
    tar -C /opt -xzf /opt/bigfile.tgz && \
    chown -R user:user /opt/bigfiledir

Since this all happens in one command the changes don’t get saved in an overlay layer.

My actual work was much more complicated so it took awhile to spot this, but it’s an important lesson nonetheless.

Keep your images small folks!

CoreOS + Etcd + Flannel = Pretty Cool

This is just a brief update on what I’m looking into.

I have a cluster, how do containers on different machines talk?

An easy solution is Flannel + Etcd. Here’s a simple example which will work on two machines. (I suggest using coreos-vagrant to create a two node cluster with flannel. Here’s my user-data and config.rb.)

First Machine:

CID=$(docker run -d alpine /bin/sh -c "while true; do sleep 1; done") && etcdctl set /server $(docker inspect --format '{{.NetworkSettings.IPAddress}}' $CID) && docker exec -it $CID /usr/bin/nc -l -p 9999 && docker stop --time=0 $CID && docker rm $CID

Note: Exit with Ctrl+P Control+Q (^P^Q), doing a Control+C (^C) will not stop and remove the docker container.

Second Machine

docker run --rm -it --add-host server:$(etcdctl get /server) alpine /usr/bin/nc server 9999

Now you should be able to chat between the two machines!

Meh? What back magic is this?

I’m simply using etcd to store the flannel IP of the server in the value of the /server key. On another machine we use that IP to attach netcat. The black magic is in flannel’s overlay network and etcd’s distributed key/value store.

Breaking apart the commands to the server:

CID=$(docker run -d --name server alpine /bin/sh -c "while true; do sleep 1; done")

This command sets up a docker container running the shell with a script that just sleeps. Simply running netcat doesn’t seem to give a good two-way connection in this example. The $CID part stores the docker container’s ID in a variable so I can use it later. I could have used the –name option to give a sensible name, but docker inspect doesn’t seem to accept a name.

etcdctl set /server $(docker inspect --format '{{.NetworkSettings.IPAddress}}' $CID)

Here I am setting a key in etcd so that another machine in fleet (which could be on the other side of the world) can access the flannel IP address of this server. This command makes use of the $CID variable set before (since it is random) and docker inspect’s format command to remove the formatting.

docker exec -it $CID /usr/bin/nc -l -p 9999

Executing netcat here instead of in the docker run phase is just for demonstration, normally this part would be done there. Because we are people who like to see things work, executing this command allows us to interact with the netcat server.

docker stop --time=0 $CID

Stopping the container with a zero time is analogous to killing it.

docker rm $CID

Get rid of that hanging container!

Lesson learned:

  1. Flannel works nicely
  2. Etcd is great
  3. Netcat is amazing
  4. Alpine is a wonderful ~5MiB distro for docker containers
  5. This stuff really needs a web UI! (If you know of a good one, leave a comment and I’ll blog about it.)

Deis … when PaaS becomes IaaS (for one tenant)

Deis - Bare Metal Cluster Diagram

Creating a bare metal clustered Platform as a Service

What will this get you? A cluster which auto-boots and provisions new nodes over PXE (network boot). You will still need to manually provision the first one, but after that all you need is to plug in new machines.

  • 1 computer for hosting the networking business
  • 1 computer to act as the bootstrapping node
  • 2 or more computers to make the cluster complete
  • 1 switch big enough to connect all the computers (plus the one you intend to use on that network)

Firewall / Router / DHCP / Kitchen sink…

Setting up a server to handle all the network tasks is a bit of an oversimplification but it works well for me, here’s how I did it:

  1. Install pfSense on a computer with at least two network adapters
  2. Configure one for WAN (your regular network)
  3. Configure the other for LAN (pick a range, I chose
  4. Login to the web admin page
  5. Go to System -> Packages
    1. Install [Filer, haproxy-devel, and TFTP]
  6. Go to Services -> DHCP Server
    1. Enable DHCP server
    2. Range –
    3. DNS servers (
    4. Domain name (whatever, you can use something like example.com)
    5. TFTP server
    6. Enable network booting
      1. next-server
      2. default bios filename pxelinux.0
  7. Services -> TFTP
    1. TFTP Daemon Interfaces: LAN
    2. Download the syslinux package from kernel.org
    3. Upload the following files to /tftpboot from the syslinux archive you downloaded (you will need to search for them though!) [pxelinux.0 ldlinux.c32 menu.c32 libutil.c32]
    4. Upload the CoreOS PXE boot files coreos_production_pxe.vmlinuz and coreos_production_pxe_image.cpio.gz (refer to the CoreOS PXE Boot Guide if you have trouble) to /tftpboot
    5. Use a client like Filezilla or SCP to upload the pxelinux.cfg/default to /tftpboot as the web UI can’t make directories. (Here’s mine, you will need to rename it to just default no extension)
  8. Diagnostics -> Filer
    1. Create three files:
      1. deis-node-auto-install.yml (remove .txt if you download it)
      2. deis-master-1.yml  (remove .txt if you download it)
      3. deis-node.yml  (remove .txt if you download it)
    2. For each of the files replace the [replace me] section with your public key
      1. You did make a public / private key pair right? No?
        1. ssh-keygen -q -t rsa -f ~/.ssh/deis -N ” -C deis
        2. copy the ssh-rsa and string into those files from ~/.ssh/deis.key
        3. also, you might need to run chmod 0700 ~/.ssh/deis since the permissions may be wrong
  9. Firewall -> Virtual IPs
    1. Add
      1. Type IP Alias
      2. Interface LAN
      3. IP Address(es) Type: Single address Address:
      4. Description Deis HA Proxy
  10. Services -> DNS Forwarder
    1. Enable!
    2. Register DHCP leases!
    3. Register DHCP static mappings!
    4. Interfaces: All
    5. Advanced
      1. address=/.example.com/
    6. Save!
  11. Services -> HA Proxy
    1. Backend
      1. Name deis_http
      2. Add (You will need to add an entry for each computer you add, so you’ll be back here later to add more, for now we will just add the one we know about.)
        1. Name: controller1
        2. Address:
        3. Port: 80
      3. Health check method: HTTP
      4. Http check URI: /health-check
      5. Connection timeout: 2147483647
      6. Server timeout: 2147483647
      7. Save!
    2. Backend (another one!) You can copy the one above with slight changes
      1. Name: deis_ssh
      2. For each server change the port to 2222
      3. Balance: Least Connections
      4. Health check method: Basic
      5. Save!
    3. Frontend
      1. Name: deis_http
      2. External address: (Deis HAProxy)
      3. Port 80
      4. Backend server pool: deis)http
      5. Type: HTTP/HTTPS(offloading)
      6. Client timeout: 2147483647
      7. Use ‘forwardfor’ option: checked!
      8. Save!
    4. Frontend (again, and yes you can copy with slight changes)
      1. Name: deis_ssh
      2. Port: 2222
      3. Backend server pool: deis_ssh
      4. Type: TCP
      5. Save!

I bet you’re done with pfSense by now. Me too. But you’re almost done!! The rest is pretty easy.

Faking out CoreOS (speeds up installs on multiple machines)

  1. Attach with Filezilla or another client to and login as root / pfsense (unless you changed the password then use that one)
  2. Change to /usr/local/www
  3. Create a directory called current
  4. Upload the following two files:
    1. coreos_production_image.bin.bz2
    2. coreos_production_image.bin.bz2.sig

Note: If you want (and you will) to add VPN, save yourself some massive headaches and just follow this guide.

Your first node… (aww how cute)

Boot your first machine with the network card and you should see a boot menu appear with two options. Pick “Live Deis CoreOS Node (Master #1)” When it’s booted into the console issue the following commands:

  1. curl > config
  2. sudo coreos-install -d /dev/sda -b -c config -V current
  3. sudo reboot

Your first node is now ready!

To N nodes and beyond!

Now just allow the subsequent machines to boot in PXE mode and the default option will automatically install and reboot your machine into the cluster.

Did you survive? Did I make some omission? Did you notice the totally bogus way I handled the network?

Post a comment! I’d love to hear from you!

The Seven Layer Cake of Web Development


Modern web development (especially for businesses) can be quite challenging and exciting! But to build a scalable  and maintainable complex web application can be a challenge. Lo and behold that it can be as simple as making a seven layer cake (yes, I know the picture has six layers, but stick with me).

Persistent Storage Layer

These days pretty much everything in a website is dictated by or stored in a persistent data storage device (SQL, NoSql, File, etc…) and the data within ranges from simple key-value pairs to (in some cases) entire web pages. It’s an important aspect of web development and shouldn’t be overlooked and glossed over.


Choose a relational database management system (T-SQL, MySQL, etc…) when you have data that needs to be related but stored separately. This is the first (and only) stop for many folks since there is a lot of expertise in the field available and RDBMS’s are pretty easily understood. Additionally there are some great tools for managing them, and the ones that are out there are generally well-supported and mature products.


Use NoSql when you require speed and simplicity. There are a tremendous number of these and they vary from simple key-value engines that run in memory with a persistent backing such as Redis to document stores like MongoDB to almost RDBMS offerings like Cassandra. Check out the Wikipedia article for more information on what’s out there.

Object Model / Data Abstraction Layer

Although many would argue against an object model, having some layer that abstracts the model of the database away from the actual usage of it is extremely useful later in the product life cycle if you discover that your chosen persistent storage provider is unable to keep up with your exploding business! Keep this layer thin and you’ll rarely need to touch it, and later if you need to swap it out you can do so quickly.

Data abstraction is another “nice to have” the idea being to keep your work atomic (simple calls that have reliable results) which can be pieced together by your business logic or controller to have the desired effect.

Business Logic Layer

Some web sites choose to combine the business logic layer with the controller which can be a mistake that later leads to large files full of spaghetti code. The business logic layer is the one that “the suits” care about the most, here is where they will tweak the rules and make crazy changes. Keep those crazy changes away from the rest of your code and you will be a very happy person.

Controller Layer

This is where it all comes together, the business logic layer uses data provided by the ORM / DAL (which in turn gets its data from the persistent storage) to deliver data to a hungry view layer. At this point it becomes easy to see how building these layers nice and separate from each other reduces code, bugs, and time spent debugging.

View Layer

In the view layer all the structure of how the data is displayed is defined. Remember to keep this layer mostly free of code, you are focusing on one thing here: interacting with the user. Ask yourself this question: “does what I’m writing affect the user’s experience with my application?” Does it have to do with colors, boxes, lines, pictures? That’s the view layer! Does it have to do with how something is stored in a database? Wrong spot! There’s some ambiguity when thinking about input validation, the simple rule is: do input validation as best you can, but if it degrades the user experience then change your approach!

Presentation Layer

The presentation layer is all about how everything comes together, the visual appeal of your software and how it interacts with the user. This is distinct from the view layer because there is no code here at all. That’s right, no HTML CSS JS or any other letters put together. How does it look, feel, smell, taste, whatever you are trying to present to the user give it good attention because I’m sure Myspace had some great code behind it, but it was severely lacking in presentation.

 Client Layer

Okay, we’re back from touch-y feel-y presentation layer back to hard code. The client layer is the stuff that’s running on the actual client machine. We’re talking HTTP headers, cookies, JavaScript, ajax, the works! Many modern browsers like Google Chrome will actually analyse your page and give advise on how to better optimize the client experience.  This includes potentially using a static content server to serve out files that rarely change (JavaScript, CSS, etc…) and handling cookies correctly. The client layer is big and stretches from network performance to JavaScript optimization to CSS fixes for the major browsers and which version they are at. Do not underestimate this as a layer and consider  methods for segregating it from the view layer both during development and deployment.

Have Fun!

In conclusion web development when done correctly can be a tremendous amount of fun, so enjoy yourself! Remember that there are many paths to the same end, but some planning ahead of time can really make you life much easier!