The declarative nature of a BOSH manifest can give a team solid confidence about what is actually running in production. Give BOSH a deployment manifest and BOSH will make it happen.

But if the manifest is 3000 lines long, and you have three of them – sandbox, staging and production – your solid confidence can evapourate.

"Exactly what is running on production and what is on staging?"

Spiff

Spiff is a CLI tool that lets you compose smaller YAML templates together into the final large BOSH deployment manifest.

One way to install spiff is via the latest traveling-bosh installer release.

curl -k -s https://raw.githubusercontent.com/cloudfoundry-community/bosh_cli_install/master/binscripts/installer | bash

Comprehension

The benefit comes from authoring your spiff templates to be easily comprehensible from the title of the template.

For example, these templates for a Cloud Foundry deployment immediately indicate what concept they are describing to a team:

• tiny/cf-tiny-dev.yml
• cf/cf-infrastructure-aws.yml
• tiny/cf-blobstore-nfs.yml
• tiny/cf-lb-haproxy.yml
• cf-no-ssl.yml
• cf-simple-secrets.yml
• cf-networking.yml

At a glance we can see: It’s a tiny deployment of Cloud Foundry, targeting AWS, using NFS for the blobstore, and HAProxy for the load balancer, but without any SSL certificates.

That’s one reason you want to use spiff. Comprehension.

Modularity

From the list of templates above you can immediately imagine interchanging them for alternate configuration.

• cf-no-ssl.yml could change with cf-self-sert-ssl.yml or cf-ssl.yml
• tiny/cf-blobstore-nfs.yml could change with tiny/cf-blobstore-s3.yml to use AWS S3 for blobstores

Spiff templates can provide modularity, which can allow you to compose your final BOSH deployments differently without much fuss.

Spiff Merge

To compose templates together you use the spiff merge subcommand.

spiff merge \
  tiny/cf-tiny-dev.yml \
  cf/cf-infrastructure-aws.yml \
  tiny/cf-blobstore-nfs.yml \
  tiny/cf-lb-haproxy.yml \
  cf-no-ssl.yml \
  cf-simple-secrets.yml \
  cf-networking.yml \
  production-stub.yml

This would output a full BOSH manifest.

All the YAML files are templates, except the last file. This is the "stub".

The stub allows you to override aspects of the templates; and to provide missing values. If you forget to provide a missing value, spiff merge will fail.

Pipelines

The stub allows you to reuse the same templates for multiple deployments – such as your CI pipeline – but with a few deployment-specific changes.

For example: For a sandbox deployment, to different AWS VPC networking and perhaps with non-production sizing, you would have a stub sandbox-stub.yml.

How to write templates

Some BOSH releases – such as Cloud Foundry’s cf-release include a set of templates: https://github.com/cloudfoundry/cf-release/tree/master/templates. But you might find they are not modular or comprehesible enough for your team.

Other BOSH releases may not include any spiff templates at all.

In either case you will want to know how to write spiff templates to suit your team. Don’t feel locked into any upstream templates.

A spiff template and a spiff stub are just a YAML files.

Given a stub sandbox-stub.yml:

name: sandbox-cf
director_uuid: 21a7a7ee-d7f1-11e4-9dd5
compilation:
  cloud_properties:
    instance_type: c1.medium

And a template cf-deployment.yml:

name: my-cf
director_uuid: (( merge ))
compilation:
  workers: 6
  cloud_properties: (( merge ))

The result is:

$ spiff merge cf-deployment.yml sandbox-stub.yml
compilation:
  cloud_properties:
    instance_type: c1.medium
  workers: 6
director_uuid: 21a7a7ee-d7f1-11e4-9dd5
name: sandbox-cf

What happened?

1. The (( merge )) token was replaced by the director_uuid value from the stub.
2. The key name was overridden by the stub’s own value sandbox-cf
3. The key compilation and its subkey compilation.workers was unaffected and included in the final YAML
4. The keys are all reordered alphabetically. Don’t know why it does that. I liked my ordering.

Merge errors

If our stub did not provide overrides for all (( merge )), then spiff merge will fail.

If our stub is only the name:

name: sandbox-cf

We get the helpful errors:

$ spiff merge cf-deployment.yml sandbox-stub.yml
2015/03/31 15:13:33 error generating manifest: unresolved nodes:
	(( merge ))	in cf-deployment.yml	compilation.cloud_properties
	(( merge ))	in cf-deployment.yml	director_uuid

This error tells us that the cf-deployment.yml template requires two keys to be provided by a stub; or another template: compilation.cloud_properties and director_uuid.

Multiple templates

compilation.cloud_properties looks like its ripe for a modular solution. If we’re on AWS then we’ll use one answer; and on bosh-lite we’ll use another.

Template cf-infra-aws.yml:

compilation:
  cloud_properties:
    instance_type: c1.medium

Template cf-infra-bosh-lite.yml:

compilation:
  cloud_properties:
    dummy: here

And we’ll put director_uuid back in the per-deployment stub:

name: sandbox-cf
director_uuid: 21a7a7ee-d7f1-11e4-9dd5

We can now create a manifest for bosh-lite or AWS:

$ spiff merge cf-deployment.yml cf-infra-bosh-lite.yml sandbox-stub.yml
compilation:
  cloud_properties:
    dummy: here
  workers: 6
director_uuid: 21a7a7ee-d7f1-11e4-9dd5
name: sandbox-cf

meta: convention for stub inputs

As spiff was being developed and simultaneously being adopted by the cf-release project, a convention evolved for describing the inputs to your templates: top-level meta key.

This also becomes a clean way to describe defaults in templates that are used in many locations.

It’s also very useful for allowing you to override an entire list.

Let’s specify the list of releases to be used by a deployment in the cf-deployment.yml:

meta:
  releases:
  - name: cf
    version: latest
name: my-cf
director_uuid: (( merge ))
releases: (( meta.releases ))
compilation:
  workers: 6
  cloud_properties: (( merge ))

The snippet (( meta.releases )) means to copy in the value from meta.releases in the same template.

You could optionally now override the entire releases array in the stub:

name: sandbox-cf
director_uuid: 21a7a7ee-d7f1-11e4-9dd5
releases:
- name: cf
  version: 205
- name: monitor-server
  version: latest

Examples

Each of the examples above are available in a GitHub repo:

git clone https://github.com/cloudfoundry-community/using-spiff-examples.git
cd using-spiff-examples
./01-simple-merge/make_manifest.sh
./02-missing-fields/make_manifest.sh
./03-modular-infra/make_manifest.sh bosh-lite
./04-meta-releases/make_manifest.sh aws

More usage

The README for spiff merge includes many other options for creating composable, modular spiff templates under Usage.