AWS, and vSphere, and OpenStack.. Oh My!!!!
Lately, we’ve been playing with a feature that has us pretty excited. Multi-CPI BOSH has become a reality and we couldn’t wait to take it for test drive. With this feature we can target multiple and disparate IaaS from a single BOSH environment.
Show Me, Show Me, Show Me, How you do that trick?!
The basic process for adding Multiple CPI’s to a single BOSH environment is as follows:
- Ensure you are on a BOSH release with adequate support (263+)
- Add an additional CPI Config to override the default
bosh update-cpi-config - Update Availability Zones in Cloud Config
bosh update-cloud-config - Upload and fix stemcells
bosh upload-stemcell - Assign appropriate AZ’s to your deployment
- Profit?
Which BOSH Version is Best?
Support was officially added in BOSH v261 but as of this writing we find the best experience to be with v263 or greater. If you keep tabs on releases you’ll see that CPI related work is common in the release notes. If you run in to any issues, it’s always a good idea to test with the latest release.
You’ll also need a recent BOSH CLI version. Learn more and find installation instructions at https://bosh.io/docs/cli-v2.html and https://apt.starkandwayne.com.
Another Config to Manage? Bah, Humbug!
This feature is possible because of the great work that has been done to separate concerns between manifest files. More deployment manifest files to manage is a feature, and it’s worth it.
We start by updating the CPI Config. For the most part, the config definition is identical to how it looked before, it’s just in a new location. One interesting caveat we ran into is that if your new CPI config is wrong, in some cases your deployment can fall back to the default config, making troubleshooting difficult. Also, there are some minor inconsistencies in the CPI config schema. For example, we had to change the address: key to host: when we moved our vSphere config in to the new scheme.
As of this writing, you can find OpenStack and vSphere examples in the CPI-Config documentation and we’ve tested both of these example schemas.
So work out your CPI config based on the docs and upload to your BOSH environment.
bosh update-cpi-config cpi-config.yml
Avail Ourselves in CPI Supremacy!
Now that we have multiple CPI’s defined, we need to assign them to availability zones; this is the magic that makes it all work. You simply supply the new CPI to the AZ definition in your cloud-config.
azs:
- name: z1
cpi: my-vsphere-cpi-name
- name: z2
cpi: m-openstack-cpi-name
cloud_properties:
availability_zone: actual-openstack-az-name
And update your cloud config.
bosh update-cloud-config my-cloud-config.yml
A Note About Networking.
Your cloud config will probably get a bit more complicated. For the simplest "hello world" deployment across multiple IaaS you’ll need to provide network configurations in cloud config that your target IaaS is configured to support. For testing more complicated situations, such as spreading deployments across multiple IaaS flavors, you may need to provide single networking configurations that can be supported by both IaaS.
Don’t Dread deprecated Deployments!
Keep in mind that every time you update your cloud config, all of the deployments on that director will become outdated; even if you’ve not changed a configuration relevant to that deployment. You’ll have to decide how you want to manage this state change problem. Currently, toggling the deployment state back to normal requires a redeployment.
However, if you haven’t changed anything that deployment cares about, like adding a CPI config to be used by a different deployment, you can decide if ignoring the state change until next deployment is a reasonable compromise.
Stemcells Schmemcells!
Now that we’ve updated our BOSH environment to be aware of more than one IaaS, the first difference we’ll notice is that there is a CPI column in the output of bosh stemcells. In order to make your first deployment, you’ll need to ensure that your stemcell/CPI pairings are in order.
Stemcells are only assigned to a CPI when they are uploaded, so if you’ve already uploaded stemcells, you’ll need to re-upload and fix them with --fix.
bosh upload-stemcell https://url/to/stemcell.tgz --fix
If you haven’t yet uploaded any stemcells, you can just upload them normally and the CPI will get assigned.
Ready to Rumble!
So theoretically, we now have a BOSH environment that can target more than one IaaS endpoint and provision a deployment with the appropriate CPI and stemcell. The deployment itself doesn’t change, just tell it which availability zones to use.
Great! But, Why Would I Want to do This?
As of this writing, we’ve been able to use this feature to deploy distinct deployments from a single BOSH across multiple IaaS. In our testing, targeting an OpenStack and a vSphere from a single BOSH environment worked fairly well. What we did not try, was spreading a single, multiple AZ BOSH deployment across multiple IaaS flavors. However, this is something that should be possible, as long as your networking is set up correctly, and we are looking forward to playing with that idea more.
Our first use for the feature will probably be incorporating in to our Proto-BOSH paradigm, this is the inception BOSH environment that we use to bring up other BOSH environments.
Just the Docs Ma’am!
A discussion about the ability to target multiple IaaS from a single BOSH environment first showed up in the
bosh-notes repository and that has some good thoughts on the history and intended use of the feature. Most other details can be found in the BOSH documentation for CPI-Config and Cloud-Config
What’s Next?
We are looking forward to seeing where this feature can take our deployment paradigms and best practices. In its current form we think it will be very useful. If you get some time to play with it, let us know in the comments below.
Spread the word
