This week, I’ve been tasked with making a change to one of our existing Concourse pipelines. This got me to thinking: A CI pipeline should be treated as if it was versioned code. If I want to change it, I should have a place to try changes out, well out of the way of the developers trying to use said pipeline to get their patches out.
"Well gee", you might say – "why don’t you just copy the pipeline, make the requisite changes, and name it something else?" Yeah, I could – but using my own Concourse would be one more step to keep me from being one fat-fingered "fly" command from plunging a digital backhoe into the dev pipeline (beeeep beeeep beeeep CRUNCH). Plus, I hadn’t stood up Concourse before, so I wanted to learn.
I stood up Concourse locally in bosh-lite using Vagrant, and it seemed to work well. I checked out concourse.ci for other ways to stand it up. There are good example manifests, but I wanted to use bosh-lite running on AWS, as I didn’t want to incur the cost of 3 VM’s just to run a dev Concourse on AWS for a day or two.
Here’s how I did it:
Deploy bosh-lite
First, I installed boss-lite, which is a great way to manage your bosh-lites, plus it makes deployment of bosh-lite easier.
Before using boss-lite, make sure you first fill out your boss-lite/bosh-lites/.envrc file. It should already be stubbed out – you just need to ensure your AWS key, ssh key, keypair, security group id, and subnet id are filled in.
To create a new bosh-lite, you simply run the following command in your boss-lite/bosh-lites directory:
$ bin/boss-lite new concourse
Once boss-lite creates your new bosh-lite, target it:
$ bosh target <your bosh-lite IP here>
Upload The Stemcell and Releases
We will need to upload releases for both Concourse and Garden. Note that from Concourse’s Github Page, releases of Concourse and Garden are specifically matched, so use those version numbers in place of the numbers below.
$ export CONCOURSE_VERSION=0.65.1
$ export GARDEN_VERSION=0.307.0
$ bosh upload release https://github.com/concourse/concourse/releases/download/v$CONCOURSE_VERSION/concourse-$CONCOURSE_VERSION.tgz
$ bosh upload release https://github.com/concourse/concourse/releases/download/v$CONCOURSE_VERSION/garden-linux-$GARDEN_VERSION.tgz
Next, upload the stemcell:
$ bosh upload stemcell https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent
Create Your Manifest
While the folks at concourse.io give a pretty good sample manifest for deploying to bosh-lite, I had to give it a few tweaks. In addition to having some problems with IP addresses (which may or may not have resulted from me using the wrong version of Garden), I also followed Dr. Nic’s blog post on how to notify our team on slack about concourse errors. My sample manifest is in this gist for anyone who wants to use it.
Deploy
Now that we have a manifest, let’s deploy:
$ bosh deployment concourse-aws-boshlite.yml
$ bosh deploy
If all goes well, you now have concourse running! Stay tuned though, there is one more step.
Make It Accessible
We need the ability to get to the Concourse API, as well as the UI. For that, we need to run a couple iptables commands directly on the bosh-lite.
First, we can use boss-lite to SSH into our bosh-lite:
$ boss-lite ssh concourse
Next, run the following two commands. NOTE: If you changed the web interface IP in your manifest, you will need to adjust these commands accordingly.
$ sudo iptables -A FORWARD -p tcp -d 10.244.8.2 --dport 8080 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
$ sudo iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 8080 -j DNAT --to-destination 10.244.8.2:8080
Now, you should be able to hit your new Concourse from the web browser on your bosh-lite IP on port 8080.
Troubleshooting Notes
I’ve found a couple times where I had to actually re-deploy my bosh-lite, due to some Garden containers that fell over and could not be killed (again – I may have been using the wrong version of Garden at the time – make sure you download the correct versions). This was evident from errors that look like this, which indicate there is some rogue Garden container using an IP address that your deployment wants to use.
Error 100: Creating VM with agent ID 'a6cebe66-873d-41a6-99cd-48f14da55088': Creating container: network already acquired: 10.244.8.8/30
If you have deleted your deployment and still suspect something is amiss, you can actually hit the Garden API like this:
$ curl http://<your bosh-lite IP here>:7777/containers
You should get output like this if you have an existing deployment:
{"handles":["c651f731-fd25-4a4e-6877-51c99136f43e","af12d63b-7159-4798-6dc1-9d4beb54bc4d","dd009585-f5dc-428f-4763-00113eb7d03f"]}
If you don’t get empty brackets, and you don’t have any deployments in BOSH, then you probably need to redeploy bosh-lite. Yes – Garden does have a DELETE endpoint, but I have not yet had it work for me.
For more information on the Garden API, go here.
Spread the word
