BUCC supports Backup & Restore finally!

Many people have asked about how to backup and restore BUCC (an introduction to BUCC can be found here. The wait is over, because as of v0.4.0, there is full support for BBR (BOSH Backup & Restore). The technical details will be given at the end of this blogpost, but first, here is the short version:

Backup

The BUCC CLI has a new wrapper command (bucc bbr) which wraps bbr director and configures the SSH key, user, and IP. So, all that’s left to do is run a backup:

bucc bbr backup

This will result in a timestamped directory in your current path with the backup artifacts. To store backups securely we recommend using SHIELD (out of scope for this blogpost), which has support for BBR .

Restore

Doing a full restore in the case of a disaster takes two steps:

# 1. creating a fresh BUCC with the state/creds.yml from backup
cd bucc
tar -xf ${backup_dir}/bosh-0-bucc-creds.tar -C state
bucc up

# 2. Restore databases and BOSH blobstore using bbr
bucc bbr restore --artifact-path=${backup_dir}

That’s it! Your BUCC should be the same as it was when the backup was taken.

Technical Details

Why did it take a bit longer than expected to add BBR support? First, BBR support has only recently been worked on by the upstream BOSH releases used by BUCC. And not all of them work well with bosh create-env. Or work with a recent Postgres version.

That’s why we had to create bucc-bbr-boshrelease to bundle all of the workarounds and to keep track of the upstream issues. Our hope is to get rid of these workarounds when the upstream issues are resolved. To ensure our workarounds work, the above backup and restore steps are actually tested as part of our pipeline.

In addition to the workarounds, we also had to make sure to backup the credentials which are generated by the BOSH CLI (–vars-store). For example, losing the CredHub encryption key and NATS CA Certificate would result in undecryptable credentials and be unable to communicate with BOSH agents.

The solution came in the form of a bucc-creds BOSH job and an accompanying ops-file. This ops-file is automatically updated by our pipeline to ensure it stays in sync with state/creds.yml.