Introducing Spruce – A More Intuitive Spiff

If you’ve been in the BOSH/CloudFoundry community for more than 10 minutes, you’ve probably heard of, seen and even used spiff. But, if you haven’t heard of it, Spiff helps to make BOSH manifests easier to manage by merging smaller template files with environment specific template files to get the final manifest.

Unfortunately, Spiff has a number of quirks and oddities in the way it behaves that may not be the most intuitive when you sit down and try to just start using it. It has a steep learning curve. It also has some idiosyncrasies that make it easy to accidentally create templates that are difficult to override, which results in end-users forking upstream templates or cargo-culting huge manifest files from deployment to deployment, struggling to manage them.

I’ve been bothered by this for a while and finally decided to do something about it. I’m pleased to tell you that Spruce and ready to help you with some of the more basic BOSH templates you create. More features are currently under development to round it out and simplify the whole manifest development + management workflow.

The goal with Spruce was to make it as intuitive as possible, while still retaining the feature set needed to manage BOSH templates. As a result there are a number of things Spruce that are a bit different than the Spiff way, but I hope they result in a more approachable process. If not, feel free to offer your suggestions.

How Do I Spruce Up My templates?

1. Get an appropriate binary for your architecture here!
2. Create your yaml files.
3. spruce merge file1.yml file2.yml ... fileN.yml > my_manifest.yml
4. Celebrate by grabbing yourself a tasty beverage!

Can I use this with my BOSH release?

Absolutely! Spruce supports a lot of the same functionality Spiff does. However, since one of its design goals was simplicity and transparency, templates built for Spiff will likely either not work without changes, or not give you the expected output.

The good news is that Spruce is very easy to adapt/write templates for! To help get you acclimated to Spruce, here’s a not-so-brief summary of its behavior:

1. Spruce starts with the left-most file, and merges each subsequent file on top of it, overriding previously seen values with the most recently seen value.
2. Spruce recursively merges maps by default.
3. Spruce’s output will include any keys defined at any point in the merging process (not just the keys defined in the last file of the merge).
4. Spruce will merge arrays by default, based on the name key of the array elements (if elements are not maps, it falls back to an index-based merge). There is also support for a variety of [array manipulations]((https://github.com/geofffranks/spruce#what-about-arrays).
5. (( param "My error message" )) is a handy syntax for requiring someone to provide a value in a downstream template.
6. (( grab properties.this.that )) is how spruce pulls in values from other parts of the datastructure. This is done after all merging, so you don’t need to worry about having the property defined when the file doing the grabbing is merged in.
7. You can grab multiple values, and their results will be turned into an array. If you grabbed any arrays, the elements of that array will be grabbed, joined with the other values, and returned as a one-level array. This is useful for things like (( grab jobs.myJob_z1.networks.myNet2.static_ips jobs.MyJob_z2.networks.myNet2.static_ips ))
8. To concatenate, use (( concat value.to.reference "literal string" another.value ))
9. You can use the --prune option to delete arbitrary keys from the final manifest, in case there are values you don’t want in the merge output (like meta).
10. (( static_ips(x, y, z) )) works the same way it does for Spiff, but has improved error messaging to help track down why your IPs aren’t getting generated properly.
11. --debug will give you step-by-step debugging of file merges, and post-processing of the yaml, to make it easy to trobuleshoot if you don’t get the results you want.

Enough Talk – I Want an Example:

Check out my in-depth conversion of cf-boshworkspace’s cf-secrets.yml from spiff-to-spruce.

The Future of Spruce

We’ve used Spruce successfully with a few BOSH deployments, as well as with Concourse pipelines. It has all the features you need for managing your BOSH manifests. However, there are a few tools missing from the spruce workflow to make it dead-simple for converting existing BOSH manifests, using Spruce with a Spiff-based BOSH release, and managing changes to deployments. Be on the lookout for spruce-gen and spruce-watch!