Stark & Wayne

Pre-Flight Checks: Sprucing up Concourse with Test Concourse

As part of a project, a client wants to have a self-deploying Concourse. Basically that means that once everything is set up, the alpha Concourse will deploy the beta Concourse and, if that completes successfully, the beta Concourse will then update/deploy the alpha Concourse. Because automation is shiny.

Current Goal

Ensure that the the beta ("test") Concourse mirrors the alpha ("production") Concourse.

What This Means

Basically, both Concourses need to be configured identically except for the necessary deviations for networking.

Important Note

By the way, since the client is using vSphere I am building this off Concourse's vsphere.yml example manifest.

CDC Pipeline: Concourse Deploys Concourse

Sprucify

For efficiency, I spruced the concourse.yml manifest. What is spruce?

A quick note about spruce

spruce is a CLI tool primarily being developed by Geoff Franks. The goal is to have spruce be the next generation replacement for spiff, which is the current tool used to generate BOSH manifests. Here's its inaugural blog post and here is where the code is located on Github.

A Quick How To

First things first: I defined the static IPs in the concourse network like this:

static:
- x.x.x.50 - x.x.x.60

Now with spruce I can reference the IP address x.x.x.50 as static_ips(0) in the concourse network, likewise x.x.x.51 is static_ips(1), etc. So this:

networks:
  - name: concourse
    static_ips: x.x.x.50

Is the same as this:

networks:
  - name: concourse
    static_ips: (( static_ips(0) ))

Instead of using anchor syntax for the ATC credentials like the Concourse example manifest, I created meta at the top level and grabbed both values:

meta:
  atc_db_name: atc
  atc_db_role:
    name: atc
    password: ATCPASSWORD
...
properties:
  atc:
    postgresql:
      database: (( grab meta.atc_db_name ))
      role: (( grab meta.atc_db_role ))

I grabbed IP addresses referenced outside of the concourse network like so:

consul:
  agent:
    servers:
      lan: (( grab jobs.discovery.networks.[0].static_ips ))

To create alpha.yml, the manifest for the alpha Concourse's BOSH deploy, run:

spruce --prune meta alpha-concourse.yml > alpha.yml

Note that the sections in the output will be alphabetized, which can be a little disorienting if you don't expect it.

Now For The Beta Concourse

Creating the beta Concourse manifest is easy as pi(e) since I will only be changing the networking. In fact, there are only two top level keys: the name and networks. That's it!

To generate beta.yml, the manifest for the beta Concourse's BOSH deploy, run:

spruce merge --prune meta alpha-concourse.yml beta-concourse.yml > beta.yml

Order matters with spruce - the order above means that the second file overrides the first. This priority holds true no matter how many files you merge.

Some Cool Things

spruce is very helpful in this project for a couple reasons:

S&W CDC Templates

In our CDC repository I have added: