This article follows along from our series of articles on deploying BOSH and Cloud Foundry using the wonderful new bosh2 CLI.

Now that you have Cloud Foundry running, you'll discover it doesn't do much without some stateful services into which you will store data. Cloud Foundry supports the Open Service Broker API to allow you to integrate any compatible service broker.

Fortunately there is one service broker that you can deploy easily with BOSH 2.0 - the Docker Broker.

We will now deploy the Docker Broker to include PostgreSQL, Redis and MySQL service offerings. It is mouth watering:

$ cf marketplace
service        plans   description
mysql56        free    MySQL 5.6 service for application development and testing
postgresql96   free    PostgreSQL 9.6 service for application development and testing
redis32        free    Redis 3.2 service for application development and testing

Quick steps

First clone the docker-broker-deployment project.

git clone https://github.com/cloudfoundry-community/docker-broker-deployment
cd docker-broker-deployment

Let's introduce a new idea - targeting your BOSH and setting your deployment name using environment variables, instead of the requiring the -e vbox -d docker-broker flags for all bosh commands.

export BOSH_ENVIRONMENT=vbox
export BOSH_DEPLOYMENT=docker-broker

Next, if you've installed the bosh2 CLI, then for the sake of future proofing this article, we will alias it to bosh (which will be its future name):

alias bosh=bosh2

It is now very simple to get the Docker Broker running:

bosh deploy docker-broker.yml --vars-store tmp/creds.yml \
  -o services/op-postgresql96.yml \
  -o services/op-redis32.yml \
  -o services/op-mysql56.yml

That's it! Up and running.

What is running? Run bosh instances to find out:

Instance                                                 Process State  AZ  IPs
docker/140b0e37-f0e1-4928-b538-b2e1a829bef9              running        z2  10.244.0.152
docker/8a3ae2e3-7843-4f3e-a0b3-79e56469267e              running        z1  10.244.0.151
docker/dde7b97e-3625-491e-815d-ecb3ab084d24              running        z3  10.244.0.153
sanity-test/69eb7045-58c2-49fc-8727-3fb1c6aed010         -              z1  10.244.0.154
subway/4bca6846-b0c3-4f84-8d34-d3872b064af9              running        z1  10.244.0.150

Three docker instances, with a front-facing subway instance which contains the Open Service Broker API interface.

If you've looked in bosh-deployment and cf-deployment repos you'll see a pattern in these three repos - one "just works" base manifest, and a collection optional modifier manifests. In this article's docker-broker-deployment repo, this base manifest is docker-broker.yml.

Does it work?

You can sanity test the deployment to confirm that each of the services is working as expected:

bosh run-errand sanity-test

When this is finished look carefully at the output. You'll see that postgresql96 and redis32 services actually self-tested - the Docker images that describe each service also know how to sanity test themselves.

For example:

Running frodenas/postgresql:9.6 sanity-test...
Waiting for postgres://ty4kuimcjem4ma9j:7pxnhxynzorbzana@10.244.0.152:32768/2yubhxqlcm5ivigo to be ready (max 60s)
10.244.0.152:32768 - accepting connections
Postgres is ready
10.244.0.152:32768 - accepting connections
+ psql postgres://ty4kuimcjem4ma9j:7pxnhxynzorbzana@10.244.0.152:32768/2yubhxqlcm5ivigo -c 'DROP TABLE IF EXISTS sanitytest;'
NOTICE:  table "sanitytest" does not exist, skipping
DROP TABLE
+ psql postgres://ty4kuimcjem4ma9j:7pxnhxynzorbzana@10.244.0.152:32768/2yubhxqlcm5ivigo -c 'CREATE TABLE sanitytest(value text);'
CREATE TABLE
+ psql postgres://ty4kuimcjem4ma9j:7pxnhxynzorbzana@10.244.0.152:32768/2yubhxqlcm5ivigo -c 'INSERT INTO sanitytest VALUES ('\''storage-test'\'');'
INSERT 0 1
+ psql postgres://ty4kuimcjem4ma9j:7pxnhxynzorbzana@10.244.0.152:32768/2yubhxqlcm5ivigo -c 'SELECT value FROM sanitytest;'
+ grep storage-test
 storage-test

Nice. Any Docker image you use in Docker Broker can also be used to self-test that the service is working as expected. See frodenas/docker-postgresql as an example.

Is it secure?

Yes. The Docker daemons are all protected by TLS certificates and the Broker API is protected by random password. In this article at no time will you personally need to see these passwords or certificates.

Easy helpful generation of certificates and passwords is one of the new features of bosh2 CLI, as well as the up-coming Credhub project.

Integration with Cloud Foundry

This article is about integrating the Docker Broker with Cloud Foundry. For that we redeploy the command above, but include the op-cf-integration.yml file and register the API as a Cloud Foundry Service Broker.

First, fetch the system_domain and the cf_admin_password data from the cf deployment:

bosh -d cf manifest > tmp/cf.yml
system_domain=$(bosh int tmp/cf.yml --path /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/system_domain)
cf_admin_password=$(bosh int tmp/cf.yml --path /instance_groups/name=uaa/jobs/name=uaa/properties/uaa/scim/users/name=admin/password)

Next, add the -o op-cf-integration.yml operator patch file, plus the additional variables required so we know into which Cloud Foundry to register our Docker Broker.

bosh deploy docker-broker.yml --vars-store tmp/creds.yml \
  -o services/op-postgresql96.yml \
  -o services/op-redis32.yml \
  -o services/op-mysql56.yml \
  -o op-cf-integration.yml \
  -v cf-api-url=https://api.$system_domain \
  -v broker-route-name=docker-broker \
  -v broker-route-uri=docker-broker.$system_domain \
  -v cf-admin-password=$cf_admin_password \
  -v cf-admin-username=admin \
  -v cf-skip-ssl-validation=true

Once completed, you can register the broker:

bosh run-errand broker-registrar

The output will include this magic:

Enabling access to all plans of service postgresql96 for all orgs as admin...
OK
Enabling access to all plans of service redis32 for all orgs as admin...
OK
Enabling access to all plans of service mysql56 for all orgs as admin...
OK
Getting service access as admin...
broker: docker-broker
   service        plan   access   orgs
   mysql56        free   all
   redis32        free   all
   postgresql96   free   all

With your cf CLI you can now see the three services ready for action:

cf marketplace

The output will as shown at the top of the article. The three -o services/op-*.yml files appear as services:

service        plans   description
mysql56        free    MySQL 5.6 service for application development and testing
postgresql96   free    PostgreSQL 9.6 service for application development and testing
redis32        free    Redis 3.2 service for application development and testing

Three brand new service offerings for everyone to use!

cf create-service postgresql96 free pg1
cf create-service-key pg1 pg1-key
cf service-key pg1 pg1-key

The output will rock your world:

Getting key pg1-key for service instance pg1 as admin...

{
 "dbname": "zvenlj1owjcyj4ot",
 "hostname": "10.244.0.152",
 "password": "8elayohhjptrfqzo",
 "port": "32768",
 "ports": {
  "5432/tcp": "32768"
 },
 "uri": "postgres://mkfqd5yzxbfqitjh:8elayohhjptrfqzo@10.244.0.152:32768/zvenlj1owjcyj4ot",
 "username": "mkfqd5yzxbfqitjh"
}

This is a PostgreSQL service with randomly generated auth credentials that is ready to use.

Backup & restore

I am repulsed by data services that do not have functioning, tested, well known backup & restore systems in place. Yuck.

But today, what you've deployed above doesn't have backup & restore. So... don't share it with anyone you love just yet.

We will introduce Shield to this deployment in a future article to provide backup & restore to each service instance.