Embed absolutely anything into BOSH releases using Docker

We need Docker images to package absolutely anything into BOSH releases, and we need to embed those Docker images inside BOSH release. The new bosh-gen v0.20 makes this very simple.

Very simple indeed:

bosh-gen package consul-image --docker-image progrium/consul

Later, on a BOSH VM, it will be available again in Docker as image progrium/consul.

An example BOSH release that packages Consul via a Docker image is https://github.com/cloudfoundry-community/consul-docker-boshrelease

CF Summit – Dr Nic and the whole Stark & Wayne team will be at CF Summit on May 11 and 12 in Santa Clara, California.

The problem

A "Good BOSH Release" contains every dependency. You can put a release tarball on a thumbdrive into a data center and you can deploy it into any BOSH.

Or you can download the release tarball from https://bosh.io/releases an know you’ll get exactly the same deployment today as you’ll get 2 years from now or 2 years ago. There is no drift.

But writing BOSH releases isn’t easy – in part because of non-BOSH dependencies; in part because of build tools, which have their own non-BOSH dependencies. And in part because of non-standard filesystem layouts. For example, Python Eggs seem to hardcode a list of paths to look up things. These are never where a BOSH release would put things.

I think Docker is an adequate workaround – you can now use any base OS, any packaging system, and package anything. Handy.

Previous BOSH releases or BOSH deployments have referenced remote Docker images on Docker Hub or private Docker registries. For example the Docker-based Cloud Foundry service broker that instantly provisioned any type of primitive service you could want.

Unfortunately until now it required that your deployment pull down whatever the latest Docker images contained. This is decoupled from the lifecyle of the BOSH releases themselves.

And this can be bad.

For example, when I first added Consul to the Docker Services project it was at v0.3.1 and I used progrium/consul image. In the default configuration templates it is still documented as v0.3.1 [sad panda link] but the upstream docker image is now packaging Consul v0.5.0. This sort of unmanaged, invisible creep could cause grief in production systems.

Another issue using upstream Docker images is security – if the upstream docker registry was hacked then you might lose control of what was running on your production system. Even with a level of uncertainty is enough to encourage the packing of the image in a BOSH release. For more on Docker security http://docs.docker.com/articles/security/#docker-daemon-attack-surface

The finally issue is that some data centers or some VMs do not allow pulling data from the Internet. If so, then you will want to package the Docker image into a BOSH release (or run your own internal Docker Registry, and populate it somehow).

We want the one enclosed tarball to describe the entire deployment

So we need Docker images but we need to embed them in our BOSH releases.

Many platform ops teams using BOSH have the above needs to flatten their use of Docker images into their BOSH universe. This blog post shows how you can do this – save Docker images into your BOSH releases; and then reuse them on your BOSH job VMs.

Create a new BOSH release

Since it was the progrium/consul docker image creep that I mentioned above, then I will demonstrate how I froze and packaged it into a BOSH release consul-docker-boshrelease.

gem install bosh-gen
bosh-gen new consul-docker
cd consul-docker-boshrelease

You are now inside a new BOSH release project and have a publicly-readable blobstore/S3 bucket ready to use.

Packaging a docker image

bosh-gen package now includes a --docker-image flag that:

• docker pull the docker image into your local docker daemon (or via boot2docker)
• docker save it into your release blobs
• creates an import job template to import it into Docker daemons in production

bosh-gen package consul-image --docker-image progrium/consul

This does all the steps above for you. It finishes by giving instructions for changes to deployment templates, etc:

docker pull progrium/consul
Pulling repository progrium/consul
e2a3eec4bab6: Download complete
511136ea3c5a: Download complete
...
a5e6aa5dc5d3: Download complete
Status: Downloaded newer image for progrium/consul:latest
docker save progrium/consul > blobs/docker-images/progrium_consul.tgz
      create  jobs/progrium_consul_image/monit
      create  jobs/progrium_consul_image/spec
      create  jobs/progrium_consul_image/templates/bin/install_ctl
      create  jobs/progrium_consul_image/templates/bin/monit_debugger
      create  jobs/progrium_consul_image/templates/helpers/ctl_setup.sh
      create  jobs/progrium_consul_image/templates/helpers/ctl_utils.sh
      create  packages/consul-image/packaging
      create  packages/consul-image/spec
Next steps:
  1. To use this BOSH release, first upload it and the docker release to your BOSH:
    bosh upload release https://bosh.io/releases/cloudfoundry-community/consul-docker
    bosh upload release https://bosh.io/d/github.com/cf-platform-eng/docker-boshrelease
  2. To use the docker image, your deployment job needs to start with the following:
    jobs:
    - name: some_job
    templates:
      # run docker daemon
      - {name: docker, release: docker}
      # warm docker image cache from bosh package
      - {name: progrium_consul_image, release: consul-docker}
  3. To simply run a single container, try the 'containers' job from 'docker' release
    https://github.com/cloudfoundry-community/consul-docker-boshrelease/blob/master/templates/jobs.yml#L18-L40

The job template progrium_consul_image will install the embedded docker image into your Docker daemon at runtime.

As such to use the Docker image your deployment manifest needs to include the following two templates for the job:

jobs:
- name: some_job
templates:
  # run docker daemon
  - {name: docker, release: docker}
  # warm docker image cache from bosh package
  - {name: progrium_consul_image, release: consul-docker}
  - {name: something-running-consul, release: your-release}

The importing of the image into Docker will happen after each VM starts up at runtime.

Playing with your Docker image

Once you have deployed your new release, plus the docker release, to a VM you can play with it:

$ bosh ssh
# sudo su -
# export PATH=$PATH:/var/vcap/packages/docker/bin
# docker --host unix:///var/vcap/sys/run/docker/docker.sock load -i /var/vcap/packages/consul-image/progrium-consul.tgz
# docker --host unix:///var/vcap/sys/run/docker/docker.sock images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
progrium/consul     latest              ac951b29fd7a        2 weeks ago         60.74 MB
# docker  --host unix:///var/vcap/sys/run/docker/docker.sock run --rm progrium/consul --version
Consul v0.5.0
Consul Protocol: 2 (Understands back to: 1)

Given that the rootfs image tarball was already on the file system it is interesting that it still takes 2 minutes to unpack the tarball and insert into local Docker image system.

Disabling internet

Inside the bosh-lite VM we can emulate a datacenter without internet:

dns_server=(`grep -Po "(?<=nameserver ).*" /etc/resolv.conf`)
sudo iptables -t nat -F w--postrouting
sudo iptables -t nat -A w--postrouting -s 10.244.0.0/19 -d $dns_server -j MASQUERADE
echo "containers do not have internet access"

Then re-deploy:

bosh delete deployment consul-containers
bosh -n deploy

If we SSH into the running container and look at its processes we see:

docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8600 -container-ip 172.17.0.1 -container-port 8600
docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8300 -container-ip 172.17.0.1 -container-port 8300
docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8500 -container-ip 172.17.0.1 -container-port 8500
/bin/consul agent -config-dir=/config --server --bootstrap-expect=1

Even without internet the Consul docker container was able to boot up on a new VM.

Summary

By embedding Docker images inside BOSH releases we solve several problems:

• Our deployments/running VMs do not need to reach out to the Internet to fetch Docker images
• We lock-step our Docker images with our BOSH releases into one release tarball
• We can now package anything, using dependencies from any packaging system thanks to Docker images

This is available now in bosh-gen v0.20.0 and higher.