Deis … when PaaS becomes IaaS (for one tenant)

Deis - Bare Metal Cluster Diagram

Creating a bare metal clustered Platform as a Service

What will this get you? A cluster which auto-boots and provisions new nodes over PXE (network boot). You will still need to manually provision the first one, but after that all you need is to plug in new machines.

  • 1 computer for hosting the networking business
  • 1 computer to act as the bootstrapping node
  • 2 or more computers to make the cluster complete
  • 1 switch big enough to connect all the computers (plus the one you intend to use on that network)

Firewall / Router / DHCP / Kitchen sink…

Setting up a server to handle all the network tasks is a bit of an oversimplification but it works well for me, here’s how I did it:

  1. Install pfSense on a computer with at least two network adapters
  2. Configure one for WAN (your regular network)
  3. Configure the other for LAN (pick a range, I chose 192.168.10.0/24)
  4. Login to the web admin page
  5. Go to System -> Packages
    1. Install [Filer, haproxy-devel, and TFTP]
  6. Go to Services -> DHCP Server
    1. Enable DHCP server
    2. Range 192.168.10.10 – 192.168.10.99
    3. DNS servers (192.168.10.1 8.8.4.4 8.8.8.8)
    4. Domain name (whatever, you can use something like example.com)
    5. TFTP server 192.168.10.1
    6. Enable network booting
      1. next-server 192.168.10.1
      2. default bios filename pxelinux.0
  7. Services -> TFTP
    1. TFTP Daemon Interfaces: LAN
    2. Download the syslinux package from kernel.org
    3. Upload the following files to /tftpboot from the syslinux archive you downloaded (you will need to search for them though!) [pxelinux.0 ldlinux.c32 menu.c32 libutil.c32]
    4. Upload the CoreOS PXE boot files coreos_production_pxe.vmlinuz and coreos_production_pxe_image.cpio.gz (refer to the CoreOS PXE Boot Guide if you have trouble) to /tftpboot
    5. Use a client like Filezilla or SCP to upload the pxelinux.cfg/default to /tftpboot as the web UI can’t make directories. (Here’s mine, you will need to rename it to just default no extension)
  8. Diagnostics -> Filer
    1. Create three files:
      1. deis-node-auto-install.yml (remove .txt if you download it)
      2. deis-master-1.yml  (remove .txt if you download it)
      3. deis-node.yml  (remove .txt if you download it)
    2. For each of the files replace the [replace me] section with your public key
      1. You did make a public / private key pair right? No?
        1. ssh-keygen -q -t rsa -f ~/.ssh/deis -N ” -C deis
        2. copy the ssh-rsa and string into those files from ~/.ssh/deis.key
        3. also, you might need to run chmod 0700 ~/.ssh/deis since the permissions may be wrong
  9. Firewall -> Virtual IPs
    1. Add
      1. Type IP Alias
      2. Interface LAN
      3. IP Address(es) Type: Single address Address: 192.168.10.2/32
      4. Description Deis HA Proxy
  10. Services -> DNS Forwarder
    1. Enable!
    2. Register DHCP leases!
    3. Register DHCP static mappings!
    4. Interfaces: All
    5. Advanced
      1. address=/.example.com/192.168.10.2
    6. Save!
  11. Services -> HA Proxy
    1. Backend
      1. Name deis_http
      2. Add (You will need to add an entry for each computer you add, so you’ll be back here later to add more, for now we will just add the one we know about.)
        1. Name: controller1
        2. Address: 192.168.10.5
        3. Port: 80
      3. Health check method: HTTP
      4. Http check URI: /health-check
      5. Connection timeout: 2147483647
      6. Server timeout: 2147483647
      7. Save!
    2. Backend (another one!) You can copy the one above with slight changes
      1. Name: deis_ssh
      2. For each server change the port to 2222
      3. Balance: Least Connections
      4. Health check method: Basic
      5. Save!
    3. Frontend
      1. Name: deis_http
      2. External address: 192.168.10.2 (Deis HAProxy)
      3. Port 80
      4. Backend server pool: deis)http
      5. Type: HTTP/HTTPS(offloading)
      6. Client timeout: 2147483647
      7. Use ‘forwardfor’ option: checked!
      8. Save!
    4. Frontend (again, and yes you can copy with slight changes)
      1. Name: deis_ssh
      2. Port: 2222
      3. Backend server pool: deis_ssh
      4. Type: TCP
      5. Save!

I bet you’re done with pfSense by now. Me too. But you’re almost done!! The rest is pretty easy.

Faking out CoreOS (speeds up installs on multiple machines)

  1. Attach with Filezilla or another client to 192.168.10.1 and login as root / pfsense (unless you changed the password then use that one)
  2. Change to /usr/local/www
  3. Create a directory called current
  4. Upload the following two files:
    1. coreos_production_image.bin.bz2
    2. coreos_production_image.bin.bz2.sig

Note: If you want (and you will) to add VPN, save yourself some massive headaches and just follow this guide.

Your first node… (aww how cute)

Boot your first machine with the network card and you should see a boot menu appear with two options. Pick “Live Deis CoreOS Node (Master #1)” When it’s booted into the console issue the following commands:

  1. curl http://192.168.10.1/deis-master-1.yml > config
  2. sudo coreos-install -d /dev/sda -b http://192.168.10.1/ -c config -V current
  3. sudo reboot

Your first node is now ready!

To N nodes and beyond!

Now just allow the subsequent machines to boot in PXE mode and the default option will automatically install and reboot your machine into the cluster.

Did you survive? Did I make some omission? Did you notice the totally bogus way I handled the network?

Post a comment! I’d love to hear from you!

Leave a Reply