Hello bosh-lite, My Old Friend

November 11, 2021

I’ve come to talk with you again

I’ve written a whole bunch of blogs about running Cloud Foundry on Kubernetes. They all work, to varying degrees of success and have the support of a smattering of communities (read: companies with large budgets + engineers + open source ideals).

Sometimes you just want BOSH. That sweet, sweet, familiar BOSH. Enter bosh-lite.

(2022-03-09 Update: Shorty after this was published, the bosh-lite maintainers switched the bosh director from being at 192.168.50.6 to 192.168.56.6, the blog was updated to reflect this)

What is `bosh-lite`?

bosh-lite works by spinning up a VM and placing BOSH on it. From there, you can use it to spin cf-deployment, zookeeper, or anything else that has a BOSH manifest.

The BOSH director is configured with runtime configs, cloud configs, stemcells, releases and all the other BOSH-like functionality you’ve come to expect.

Installation

The tool has been around forever, with a very good tutorial on installing and configuring bosh-lite located at https://bosh.io/docs/bosh-lite/. The rest of this blog assumes that you have followed these instructions up to step #7.

I had an issue with a newer version of VirtualBox (6.1.28), what worked for me is version 6.1.26 which is available here.

It ends with an example deployment of zookeeper which I guess is cool, but I’m guessing most BOSH directors are associated with a Cloud Foundry deployment. Read on for tips that maybe aren’t obvious from the documentation!

Deploying Cloud Foundry

In cf-deployment-land there is documentation for using BBL to deploy bosh-lite and CF to GCP or AWS. I point these instructions out in case you have an unlimited IAAS budget, however I’ll show you how to deploy this to your Mac.

There is also a Readme within cf-deployment For Operators Deploying CF to local bosh-lite. I’ll be using this as the basis for the scripting below.

Clone `cf-deployment`

For these examples, I’ve assumed you will clone the CF repo to your home folder:

cd ~

git clone https://github.com/cloudfoundry/cf-deployment.git

Setup BOSH Environment variables

This is spread across a few places in the documentation, I’ve gathered them all together:

export CREDHUB_SERVER=https://192.168.56.6:8844
export CREDHUB_CLIENT=credhub-admin
export CREDHUB_SECRET=$(bosh interpolate ~/deployments/vbox/creds.yml --path=/credhub_admin_client_secret)
export CREDHUB_CA_CERT="$(bosh interpolate ~/deployments/vbox/creds.yml --path=/credhub_tls/ca )"$'\n'"$( bosh interpolate ~/deployments/vbox/creds.yml --path=/uaa_ssl/ca)"

export BOSH_CLIENT=admin
export BOSH_CLIENT_SECRET="$(bosh int ~/deployments/vbox/creds.yml --path /admin_password)"
export BOSH_CA_CERT="$(bosh interpolate ~/deployments/vbox/creds.yml --path /director_ssl/ca)"
export BOSH_ENVIRONMENT=vbox

bosh alias-env vbox -e 192.168.56.6 --ca-cert <(bosh int ~/deployments/vbox/creds.yml --path /director_ssl/ca)

You might consider adding this to your bash/zsh profile.

Upload Stemcell, Cloud Config and Runtime Config

bosh upload-stemcell --sha1 f399044d2ebe3351f0f1b0b3f97ef11464d283b4 "https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-xenial-go_agent?v=621.125"
bosh update-runtime-config ~/workspace/bosh-deployment/runtime-configs/dns.yml --name dns
bosh update-cloud-config ~/cf-deployment/iaas-support/bosh-lite/cloud-config.yml

As time goes by, CF may complain of wanting a newer stemcell version, update the bosh upload-stemcell command with the requested version. The version and sha1 are listed here, note that bosh-lite uses warden stemcells.

Go Get a Drink

This part will take around 45 minutes to complete, unless you’ve changed the IP address of the BOSH director, the following command will deploy CF.

cd ~/cf-deployment
bosh -e 192.168.56.6 -d cf deploy \
  cf-deployment.yml \
  -o operations/bosh-lite.yml \
  -v system_domain=bosh-lite.com

Log Into CF

This is done in two steps, targeting the API and then logging in, pulling the admin password out of credhub without printing it to the screen:

cf api https://api.bosh-lite.com --skip-ssl-validation
cf login -u admin -p $(credhub get -n $(credhub find -n admin | grep cf_admin | cut -d: -f2) | grep value | cut -d: -f2) -o system -s test

Tada!

Create a space and start pushing apps.

(Optional) Setup SSH to BOSH Director

Why? I like to be able to SSH to my BOSH Director to poke around. I’m weird. This is not required for the CF deployment, but nice to have:

bosh int ~/deployments/vbox/creds.yml --path /jumpbox_ssh/private_key > ~/deployments/vbox/jumpbox.key
chmod 600 ~/deployments/vbox/jumpbox.key
ssh [email protected] -i ~/deployments/vbox/jumpbox.key

Beware of Dragons

Issue #1 – Reboot

Rebooting your Mac causes bosh-lite to misbehave, badly, unless you take a couple easy steps.

Before shutdown:

In Oracle VM VirtualBox Manager select the VM, right click and navigate to Close > Save State

After reboot:

In Oracle VM VirtualBox Manager select the VM, right click and navigate to Start > Headless Start

Even performing these steps I can’t always bring the BOSH Director back online.

If you have accidentally rebooted the host you can recreate the BOSH director using the state file and use bosh cck to recover the broken CF deployment:

cd ~/deployments/vbox

bosh create-env ~/workspace/bosh-deployment/bosh.yml \
  --state ./state.json \
  -o ~/workspace/bosh-deployment/virtualbox/cpi.yml \
  -o ~/workspace/bosh-deployment/virtualbox/outbound-network.yml \
  -o ~/workspace/bosh-deployment/bosh-lite.yml \
  -o ~/workspace/bosh-deployment/bosh-lite-runc.yml \
  -o ~/workspace/bosh-deployment/uaa.yml \
  -o ~/workspace/bosh-deployment/credhub.yml \
  -o ~/workspace/bosh-deployment/jumpbox-user.yml \
  --vars-store ./creds.yml \
  -v director_name=bosh-lite \
  -v internal_ip=192.168.56.6 \
  -v internal_gw=192.168.56.1 \
  -v internal_cidr=192.168.50.0/24 \
  -v outbound_network_name=NatNetwork --recreate

bosh cck

If that doesn’t work, you can completely wipe and recreate the BOSH and CF deployment by:

Logging into Oracle VM VirtualBox Manager, select the VM, right click and select Remove > Delete all files
Delete the state file at ~/deployments/vbox/state.json
Rerun the bosh create-env command then redeploy CF.

Issue 2 – Networking issues

If you get:

Deploying:
  Creating instance 'bosh/0':
    Waiting until instance is ready:
      Post "https://mbus:<redacted>@192.168.56.6:6868/agent": dial tcp 192.168.56.6:6868: connect: connection refused

Exit code 1

You probably skipped the step to add the host routing:

sudo route add -net 10.244.0.0/16     192.168.56.6

Final Thought

We all deserve nice things. bosh-lite is one of those nice things for folks who enjoy BOSH and want to use it on their own computer.

Enjoy!

Written by:
Chris Weibel

Senior Cloud Engineer