Detecting Warden in a BOSH Release

This morning, I was trying to wrangle our CI/CD pipeline for the Containers BOSH Release so that I could cut a 1.1.0 release and generally forget about the process of integration testing.

Our stock pipeline architecture for BOSH releases runs a deployment test by taking a manifest — either the example manifest or a CI-specific manifest — and deploying it to a BOSH director. If the deployment takes, the BOSH release is considered fit for a release.

To conserve space, we usually target a BOSH director running the Warden CPI, which just spins up Linux containers for each BOSH instance group. For most things, this is sufficient; BOSH releases rarely care about the infrastructure they are deployed on, after all.

But the Containers BOSH Release is a bit different. It runs Docker, which needs a whole slew of container-y things like cgroups to function. When I tried deploying it onto a containerized CPI, it blew up, as things on the cutting edge are wont to do.

Task 352 | 18:16:01 | Updating instance docker:   docker/b3ab4f49-2f05-4865-b46c-eb248ebded5b (0) (canary) (00:02:26)
   L Error:
       'docker/b3ab4f49-2f05-4865-b46c-eb248ebded5b (0)' is not running
       after update. Review logs for failed jobs: docker, compose

Looking into the logs, I found that, at least under Warden, you don’t have access to mount new things (including cgroup hierarchies) under /sys/fs/cgroup. It’s just plain not allowed.

A simple workaround is to move the mountpoint elsewhere. Docker still finds the cgroups wherever they happen to be mounted. However, since this is a workaround specifically for Warden, I was hoping to find a way to limit the behavior to only occur there. Other IaaS deployments (vSphere, AWS, GCP, etc.) are perfectly happy to mount things where they belong.

Luckily, I found just what I was looking for in /var/vcap/bosh/etc/infrastructure. On Warden, this contains the text warden, making it dead easy to recognize a Warden CPI from a mile away:

if grep -q warden /var/vcap/bosh/etc/infrastructure ]]; then
  echo 'Hello, Warden!'
  # ... do warden-y things ...
else
  # ...
fi

In the past, I’ve resorted to such tricks as the im_in_warden_dont_do_X property trick:

properties:
  im_in_warden_dont_do_nfs: yes

or the just-ignore-the-failure trick, which is less than ideal.

Now, armed with /var/vcap/bosh/etc/infrastructure, I can make IaaS-specific decisions to tailor my BOSH releases to the situation at hand.