Mar 31, 2015 Composing BOSH manifests with Spiff
The declarative nature of a BOSH manifest can give a team solid confidence about what is actually running in production. Give BOSH a deployment manifest and BOSH will make it happen.
But if the manifest is 3000 lines long, and you have three of them - sandbox, staging and production - your solid confidence can evapourate.
"Exactly what is running on production and what is on staging?"
Spiff is a CLI tool that lets you compose smaller YAML templates together into the final large BOSH deployment manifest.
One way to install
spiff is via the latest traveling-bosh installer release.
curl -k -s https://raw.githubusercontent.com/cloudfoundry-community/bosh_cli_install/master/binscripts/installer | bash
The benefit comes from authoring your spiff templates to be easily comprehensible from the title of the template.
For example, these templates for a Cloud Foundry deployment immediately indicate what concept they are describing to a team:
At a glance we can see: It's a tiny deployment of Cloud Foundry, targeting AWS, using NFS for the blobstore, and HAProxy for the load balancer, but without any SSL certificates.
That's one reason you want to use spiff. Comprehension.
From the list of templates above you can immediately imagine interchanging them for alternate configuration.
cf-no-ssl.ymlcould change with
tiny/cf-blobstore-nfs.ymlcould change with
tiny/cf-blobstore-s3.ymlto use AWS S3 for blobstores
Spiff templates can provide modularity, which can allow you to compose your final BOSH deployments differently without much fuss.
To compose templates together you use the
spiff merge subcommand.
spiff merge \ tiny/cf-tiny-dev.yml \ cf/cf-infrastructure-aws.yml \ tiny/cf-blobstore-nfs.yml \ tiny/cf-lb-haproxy.yml \ cf-no-ssl.yml \ cf-simple-secrets.yml \ cf-networking.yml \ production-stub.yml
This would output a full BOSH manifest.
All the YAML files are templates, except the last file. This is the "stub".
The stub allows you to override aspects of the templates; and to provide missing values. If you forget to provide a missing value,
spiff merge will fail.
The stub allows you to reuse the same templates for multiple deployments - such as your CI pipeline - but with a few deployment-specific changes.
For example: For a sandbox deployment, to different AWS VPC networking and perhaps with non-production sizing, you would have a stub
How to write templates
Some BOSH releases - such as Cloud Foundry's cf-release include a set of templates: https://github.com/cloudfoundry/cf-release/tree/master/templates. But you might find they are not modular or comprehesible enough for your team.
Other BOSH releases may not include any spiff templates at all.
In either case you will want to know how to write spiff templates to suit your team. Don't feel locked into any upstream templates.
A spiff template and a spiff stub are just a YAML files.
Given a stub
name: sandbox-cf director_uuid: 21a7a7ee-d7f1-11e4-9dd5 compilation: cloud_properties: instance_type: c1.medium
And a template
name: my-cf director_uuid: (( merge )) compilation: workers: 6 cloud_properties: (( merge ))
The result is:
$ spiff merge cf-deployment.yml sandbox-stub.yml compilation: cloud_properties: instance_type: c1.medium workers: 6 director_uuid: 21a7a7ee-d7f1-11e4-9dd5 name: sandbox-cf
(( merge ))token was replaced by the
director_uuidvalue from the stub.
- The key
namewas overridden by the stub's own value
- The key
compilationand its subkey
compilation.workerswas unaffected and included in the final YAML
- The keys are all reordered alphabetically. Don't know why it does that. I liked my ordering.
If our stub did not provide overrides for all
(( merge )), then
spiff merge will fail.
If our stub is only the
We get the helpful errors:
$ spiff merge cf-deployment.yml sandbox-stub.yml 2015/03/31 15:13:33 error generating manifest: unresolved nodes: (( merge )) in cf-deployment.yml compilation.cloud_properties (( merge )) in cf-deployment.yml director_uuid
This error tells us that the
cf-deployment.yml template requires two keys to be provided by a stub; or another template:
compilation.cloud_properties looks like its ripe for a modular solution. If we're on AWS then we'll use one answer; and on bosh-lite we'll use another.
compilation: cloud_properties: instance_type: c1.medium
compilation: cloud_properties: dummy: here
And we'll put
director_uuid back in the per-deployment stub:
name: sandbox-cf director_uuid: 21a7a7ee-d7f1-11e4-9dd5
We can now create a manifest for bosh-lite or AWS:
$ spiff merge cf-deployment.yml cf-infra-bosh-lite.yml sandbox-stub.yml compilation: cloud_properties: dummy: here workers: 6 director_uuid: 21a7a7ee-d7f1-11e4-9dd5 name: sandbox-cf
meta: convention for stub inputs
As spiff was being developed and simultaneously being adopted by the cf-release project, a convention evolved for describing the inputs to your templates: top-level
This also becomes a clean way to describe defaults in templates that are used in many locations.
It's also very useful for allowing you to override an entire list.
Let's specify the list of releases to be used by a deployment in the
meta: releases: - name: cf version: latest name: my-cf director_uuid: (( merge )) releases: (( meta.releases )) compilation: workers: 6 cloud_properties: (( merge ))
(( meta.releases )) means to copy in the value from
meta.releases in the same template.
You could optionally now override the entire
releases array in the stub:
name: sandbox-cf director_uuid: 21a7a7ee-d7f1-11e4-9dd5 releases: - name: cf version: 205 - name: monitor-server version: latest
Each of the examples above are available in a GitHub repo:
git clone https://github.com/cloudfoundry-community/using-spiff-examples.git cd using-spiff-examples ./01-simple-merge/make_manifest.sh ./02-missing-fields/make_manifest.sh ./03-modular-infra/make_manifest.sh bosh-lite ./04-meta-releases/make_manifest.sh aws
The README for
spiff merge includes many other options for creating composable, modular spiff templates under Usage.