Introduction

Deploying software to multiple environments (such as dev / staging / production) introduces operational complexity that requires explicit managing in order to ensure parity between environments.

Previously I wrote an article introducing how you can use cepler to significantly reduce this overhead (and it is recommended to read that first).

In this article we will use cepler-templates to automate the execution of the cepler check, cepler prepare, cepler record cycle to deploy cf-for-k8s using GitHub actions.

Prerequisites

If you want to follow along with the demo, you will need access to two publicly accessible Kubernetes clusters representing two environments that we want to deploy Cloud Foundry to.

If you want the resulting Cloud Foundry to be fully functional you will also need a DNS name you can use to access Cloud Foundry.

Preparation

First, we will clone the cf-for-k8s repo and generate some values we need.

$ git clone https://github.com/cloudfoundry/cf-for-k8s && cd cf-for-k8s$ git checkout v1.0.0
$ ./hack/generate-values.sh -d <testflight-dns> > testflight-values.yml
$ ./hack/generate-values.sh -d <staging-dns> > staging-values.yml

Then, we will create a new repository to store the files needed to deploy cf-for-k8s.

• Go to github.com and create a new repository called cf-k8s-cepler.

$ cd ..
$ git clone git@github.com:your-github-org/cf-k8s-cepler.git && cd cf-k8s-cepler

Config Files

We will use vendir to sync the files we need from cf-for-k8s:

$ mkdir k8s
$ cat <<EOF > vendir.yml
---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
minimumRequiredVersion: 0.8.0
directories:
- path: k8s
  contents:
  - path: cf-for-k8s
    git:
      url: https://github.com/cloudfoundry/cf-for-k8s
      ref: v1.0.0
    includePaths:
    - config/**/*
EOF
$ vendir sync
$ git add . && git commit -m 'Sync cf-for-k8s config files'

Now, we need to add the values we generated in the previous step and append some Docker Hub credentials to it. Don't actually add your password here, that will be injected via the GitHub secrets mechanism.

$ cp ../cf-for-k8s/testflight-values.yml ./k8s/
cat <<EOF >> k8s/testflight-values.yml
app_registry:
  hostname: https://index.docker.io/v1/
  repository_prefix: "<dockerhub_username>"
  username: "<dockerhub_username>"
  password: DUMMY
EOF
$ cp ../cf-for-k8s/staging-values.yml ./k8s/
cat <<EOF >> k8s/staging-values.yml
app_registry:
  hostname: https://index.docker.io/v1/
  repository_prefix: "<dockerhub_username>"
  username: "<dockerhub_username>"
  password: DUMMY
EOF
$ git add . && git commit -m 'Add environment-values'

Next, we will add a cepler.yml and ci.yml which are needed to generate the deployment pipeline.

$ cat <<EOF > cepler.yml
environments:
  testflight:
    latest:
    - k8s/cf-for-k8s/**/*
    - k8s/testflight-values.yml
  staging:
    passed: testflight
    propagated:
    - k8s/cf-for-k8s/**/*
    latest:
    - k8s/staging-values.yml
EOF
$ cat <<EOF > ci.yml
cepler:
  config: cepler.yml
driver:
  type: github
  repo:
    access_token: ${{ secrets.ACCESS_TOKEN }}
    branch: master
  secrets:
    app_registry:
      password: ${{ secrets.DOCKERHUB_PASSWORD }}
processor:
  type: ytt
  files:
  - k8s/cf-for-k8s/config
  - k8s/*.yml
executor:
  type: kapp
  environments:
    testflight:
      app_name: testflight-cf
      ca_cert: ${{ secrets.TESTFLIGHT_CA }}
      server: ${{ secrets.TESTFLIGHT_SERVER }}
      token: ${{ secrets.TESTFLIGHT_TOKEN }}
    staging:
      app_name: staging-cf
      ca_cert: ${{ secrets.STAGING_CA }}
      server: ${{ secrets.STAGING_SERVER }}
      token: ${{ secrets.STAGING_TOKEN }}
EOF
$ git add . && git commit -m 'Add cepler.yml and ci.yml'

The cepler.yml file configures the order in which the environments are deployed and which files belong to each environment.

The ci.yml tells the cepler-templates processor how the CD pipeline should be built. In this case, we are using github as a driver ytt as a processor and kapp as an executor.

Secrets

Now, go to https://github.com/your-github-org/cf-k8s-cepler/settings/secrets and add the following secrets:

• DOCKERHUB_PASSWORD - password to sign in to Docker Hub
• ACCESS_TOKEN - a personal access token that can push to this repository. You can create one under https://github.com/settings/tokens.
• TESTFLIGHT_SERVER - the endpoint of the testflight Kubernetes cluster.
• TESTFLIGHT_CA - the CA cert for the Kubernetes cluster you want to use for the testflight environment.
• STAGING_SERVER - the endpoint of the staging Kubernetes cluster.
• STAGING_CA - the CA cert for the Kubernetes cluster you want to use for the staging environment.

To gain access, we also need a token from a service account that has the required permissions for deploying cf:

$ kubectl config set-context <cluster>
$ kubectl apply -f https://raw.githubusercontent.com/starkandwayne/cf-k8s-cepler/master/deployer-account.yml
$ secret_name=$(kubectl get serviceaccount cf-deployer -o json | jq -r '.secrets[0].name')
$ kubectl get secrets ${secret_name} -o json | jq -r '.data.token' | base64 --decode
<token>

Copy the resulting token into the TESTFLIGHT_TOKEN and STAGING_TOKEN secrets respectively.

Configuring github-actions

Now everything is in place and we can create our continuous deployment setup:

$ workflows_dir=.github/workflows
$ mkdir -p ${workflows_dir}
$ docker run -v $(pwd):/workspace/inputs -it bodymindarts/cepler-templates:0.2.0 > ${workflows_dir}/deploy-cf-environments.yml
$ git add . && git commit -m 'Add deploy-cf-environments workflow'

Deployment

Once you have pushed your repo upstream to GitHub, an action should be kicked off:

$ git push -u origin master

You can follow the progress of the triggered workflows here: https://github.com/your-github-org/cf-k8s-cepler/actions

If you click on the latest commit you will be able to drop down the cepler-deploy workflow.

There are two jobs; deploy-testflight and deploy-staging.

The initial run of deploy-staging should fail since it depends on testflight having been completed successfully at least once. If you follow the deploy-testflight job the deployment should complete correctly and produce an updated cepler state:

$ git pull
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 1), reused 4 (delta 1), pack-reused 0
Unpacking objects: 100% (4/4), 5.83 KiB | 2.92 MiB/s, done.
From github.com:bodymindarts/cf-k8s-cepler
   b68074a..61d0273  master     -> origin/master
Updating b68074a..61d0273
Fast-forward
 .cepler/testflight.state | 566 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 566 insertions(+)
 create mode 100644 .cepler/testflight.state

This commit should in-turn trigger another run of the workflow. You can go back to https://github.com/your-github-org/cf-k8s-cepler/actions and click on the latest commit [cepler] Updated testflight state to watch the next deploy.
This time the deploy-testflight job should complete as a no-op. The deploy-staging job should complete successfully creating another commit.

From here on, any change to the files referenced in the cepler.yml file should trigger successive deploys. Hence, we have achieved a continuous deployment pipeline that deploys cf-for-k8s to successive environments via GitHub actions.

Accessing CF

To check that things are working as expected, you can find out the external IP of the istio-ingressgateway and point your DNS entry to it:

$ kubectl get svc -n istio-system
NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                                                      AGE
istio-ingressgateway   LoadBalancer   192.168.77.107   35.246.241.158   15021:31048/TCP,80:30241/TCP,443:30963/TCP,15443:30925/TCP   23m
istiod                 ClusterIP      192.168.90.225   <none>           15010/TCP,15012/TCP,443/TCP,15014/TCP,853/TCP                23m

Then once your DNS has been updated run:

$ cf api api.<testflight-dns> --skip-ssl-validation
$ cf auth admin `cat k8s/testflight-values.yml | yq -r .cf_admin_password`

Conclusion

We have demonstrated how using cepler-templates can make it very simple to pipeline a complex system like cf-for-k8s using the ci.yml and cepler.yml as configuration inputs.

The post Continuously deploy cf-for-k8s via GitHub Actions using cepler-templates appeared first on Stark & Wayne.

In this blog post we will introduce cepler a tool for managing the state of files representing a system deployed to multiple environments.

Introduction

When operating software it is common to have more than 1 deployment of your system running. These deployments are typically segregated into multiple environments (such as dev / staging / production). One goal here is to de-risk making changes to the production system where end-users may be effected. By trying out software upgrades or config changes on a 'non-production' we can verify our expectations around the changes we are about to make .

This approach is premised by the assumption that environments are identical to begin with (see 12factor.net/dev-prod-parity for more information). In practice this is almost never the case, at a minmum because the production environment will necesarrily have real-world load that the other environments won't.
Nevertheless having multiple environments can go a long way in improving the stability of your production work-loads.

Managing multiple environments does introduce some complexity and operational overhead.
Questions that require answering by ops teams are typically:

• How do we inject environment specific configuration?
• When making changes - how do we ensure an orderly propagation of the changes from 1 environment to the next?

There are various patterns and best practices for dealing with this complexity that are coupled to specific ops toolchains. Here I would like to introduce cepler as a general tooling-independant solution to this problem.

Setup

In this demo we will show how to use cepler to manage the shared and environment-specific configuration files needed to deploy software. As a trivial example we will be deploying a container running nginx to kubernetes.

First you will need access to a kubernetes - for example via minikube:

$ minikube start(...)
$ kubectl get nodes
NAME       STATUS   ROLES    AGE   VERSION
minikube   Ready    master   23d   v1.19.0

To merge config files together we will use spruce.

Cepler can be downloaded as a pre-built binary.
Or if you have a rust toolchain installed via:

$ cargo install cepler

Check that everything is installed via:

$ cepler --version
cepler 0.4.5
$ spruce --version
spruce - Version 1.27.0

Test deploy

Before introducing cepler lets clone the cepler-demo repository and have a look at what we are deploying.

$ git clone https://github.com/starkandwayne/cepler-demo && cd cepler-demo && git checkout -b demo

The following files make up the configuration of our system across all environments:

$ tree k8s
k8s
├── deployment.yml
└── environments
    ├── production.yml
    ├── shared.yml
    └── staging.yml
1 directory, 4 files

The k8s/deployment.yml file represents the 'system' we are deploying:

$ cat k8s/deployment.yml
meta:
  environment_name: (( param "Please provide meta.environment_name" ))
  app_name: nginx
  deployment_name: (( concat meta.environment_name "-" meta.app_name "-deployment" ))
  deployment_tags:
    app: (( concat meta.environment_name "-" meta.app_name ))
  image_tag: (( param "Please provide meta.image_tag" ))
apiVersion: apps/v1
kind: Deployment
metadata:
  name: (( grab meta.deployment_name ))
spec:
  selector:
    matchLabels: (( grab meta.deployment_tags ))
  replicas: 2
  template:
    metadata:
      labels: (( grab meta.deployment_tags ))
    spec:
      containers:
      - name: (( grab meta.app_name ))
        image: (( concat "nginx:" meta.image_tag ))
        ports:
        - containerPort: 80

At the top of the file under the meta tag we have deduplicated some settings and also specified some keys that require overriding:

$ spruce merge k8s/deployment.yml
2 error(s) detected:
 - $.meta.environment_name: Please provide meta.environment_name
 - $.meta.image_tag: Please provide meta.image_tag

The meta.environment_name override will be specified via an environment specific input file:

$ cat k8s/environments/staging.yml
meta:
  environment_name: staging

The meta.image_tag setting represents the version of our system that we will want to propagate from environment to environment.

$ cat k8s/environments/shared.yml
meta:
  image_tag: "1.18.0"

To deploy the staging environment we could use the following command:

$ spruce merge --prune meta k8s/*.yml k8s/environments/shared.yml k8s/environments/staging.yml | kubectl apply  -f -
deployment.apps/staging-nginx-deployment created
$ kubectl get deployments
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
staging-nginx-deployment   2/2     2            2           4s

Introducing cepler

To let cepler manage the state of the files that are specific to an environment we need to add a cepler.yml:

$ cat cepler.yml
environments:
  staging:
    latest:
    - k8s/*.yml
    - k8s/environments/shared.yml
    - k8s/environments/staging.yml
  production:
    latest:
    - k8s/environments/production.yml
    passed: staging
    propagated:
    - k8s/*.yml
    - k8s/environments/shared.yml

As we can see the cepler.yml file specifies which files make up an environment and which of those should be vetted in a previous environment.

The check command gives us feedback on wether or not files have changed in a way that requires a new deploy:

$ cepler check -e staging
File k8s/deployment.yml was added
File k8s/environments/shared.yml was added
File k8s/environments/staging.yml was added
Found new state to deploy - trigger commit f5b1ba0
$ cepler check -e production
Error: Previous environment 'staging' not deployed yet

At this point staging is ready to deploy but production shouldn't be deployed since the propagated files haven't been vetted yet.

Deploying an environment

To prepare for deploying an environment. We use the prepare command:

$ cepler prepare -e staging

In this case the command will be a no-op because for staging all files that are relevant should be checked out to their latest commited state (see cepler.yml above).
To be sure that no other files accidentally taint the configuration of the environment we are about to deploy we can add the --force-clean flag rendering only the files that pass the specified globs.

$ cepler prepare -e staging --force-clean
$ tree
.
├── cepler.yml
└── k8s
    ├── deployment.yml
    └── environments
        └── shared.yml
        └── staging.yml

Now that we have just the files we want in our workspace we can simplify the deploy command:

$ spruce merge --prune meta k8s/**/*.yml | kubectl apply  -f -

Once the deploy is complete we want cepler to record the state of the files for later reproduction or propagation. The record command will persist metadata about the state of the files involved in a deploy into a state file and commit it to the repository.

$ cepler record -e staging
Recording current state
Adding commit to repository to persist state
$ cat .cepler/staging.state
---
current:
  head_commit: 12d50cd01cf8631fb73a5ddcc52316ccae1b4988
  files:
    "{latest}/k8s/deployment.yml":
      file_hash: d78a37bbd8971a40c49841fe958d6ddb59444c36
      from_commit: f5b1ba0a92be43c038120c6fb2447df98c4df79a
      message: Readme
    "{latest}/k8s/environments/shared.yml":
      file_hash: 23451f22e83b6e8da62c2198ac43142d08f1b8f6
      from_commit: c485204c31b86d81b14ea829bdd2a5f56ac24dd8
      message: Use image tag as shared input
    "{latest}/k8s/environments/staging.yml":
      file_hash: ab94f97964beadcb829d8a749da7cff05b82d874
      from_commit: 8140c5d28607bcb33fb321acd565a4f542373e81
      message: Initial commit%

For each file we have the file hash (can be verified via git hash-object k8s/deployment.yml) and the commit hash + message of the last commit that changed the file.

Propagating changes

At this point we can re-check production and expect that it needs deploying.

$ cepler check -e production
File k8s/environments/production.yml was added
File k8s/deployment.yml was added
File k8s/environments/shared.yml was added
Found new state to deploy - trigger commit a12695c

But before deploying production lets assume that someone has checked in a later version of the system to be deployed.

$ cat <<EOF > k8s/environments/shared.yml
meta:
  image_tag: "1.19.0"
EOF
$ git add k8s/environments/shared.yml && git commit -m 'Bump app version'

At this point the state of k8s/environments/shared.yml is different from what was recorded as the last deployment to staging:

$ grep 'environments/shared.yml' -A 1 .cepler/staging.state
   "{latest}/k8s/environments/shared.yml":
     file_hash: 23451f22e83b6e8da62c2198ac43142d08f1b8f6
$ git hash-object k8s/environments/shared.yml
dfbaa85e62ce8edc7fc90c9dab106d2e2e4945ec

The latest state of the file hasn't been vetted yet (via a deploy to staging) so when we prepare the production deploy it will check out the last state to pass staging.

$ cepler prepare -e production --force-clean
WARNING removing all non-cepler specified files
$ tree
.
├── cepler.yml
└── k8s
    ├── deployment.yml
    └── environments
        ├── production.yml
        └── shared.yml
$ git hash-object k8s/environments/shared.yml
23451f22e83b6e8da62c2198ac43142d08f1b8f6

Now that we have prepared the workspace with the files for production we can go ahead and deploy to production:

$ spruce merge --prune meta k8s/**/*.yml | kubectl apply  -f -
% kubectl get deployments
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
production-nginx-deployment   2/2     2            2           78s
staging-nginx-deployment      2/2     2            2           19s

After deploying we need to record the state for production:

$ cepler record -e production
Recording current state
Adding commit to repository to persist state
$ cat .cepler/production.state
---
current:
  head_commit: 09cd76205cc1efed1a975a711f8331ba1ee9f256
  propagated_head: 12d50cd01cf8631fb73a5ddcc52316ccae1b4988
  files:
    "{latest}/k8s/environments/production.yml":
      file_hash: 8d7bae8892cb8e02d318b0829198a2b6d8efdd4e
      from_commit: d2f769d275a2e5808d6f2be6c20d4b6cd1ce3fbe
      message: Move testflight -> production
    "{staging}/k8s/deployment.yml":
      file_hash: d78a37bbd8971a40c49841fe958d6ddb59444c36
      from_commit: f5b1ba0a92be43c038120c6fb2447df98c4df79a
      message: Readme
    "{staging}/k8s/environments/shared.yml":
      file_hash: 23451f22e83b6e8da62c2198ac43142d08f1b8f6
      from_commit: c485204c31b86d81b14ea829bdd2a5f56ac24dd8
      message: Use image tag as shared input
propagated_from: staging%

Finally after checking out the head state again lets see what check returns:

$ git checkout .
$ cepler check -e production
Nothing new to deploy
$ cepler check -e staging
File k8s/environments/shared.yml changed
Found new state to deploy - trigger commit ad57826

Since we just deployed and recorded production there is nothing to do for that environment.
But we haven't yet applied the upgraded version in the shared.yml file to staging which is why that check is telling us there is a new state.
Also note that the 'trigger commit' accuratly identifies the last change that was relevent to the state of the environment:

% git show ad57826
commit ad578268492be4c520cc108cd210cf526271b7c5
Author: Justin Carter <justin@misthos.io>
Date:   Thu Oct 15 10:44:50 2020 +0200
    Bump app version
diff --git a/k8s/environments/shared.yml b/k8s/environments/shared.yml
index 23451f2..dfbaa85 100644
--- a/k8s/environments/shared.yml
+++ b/k8s/environments/shared.yml
@@ -1,2 +1,2 @@
meta:
-  image_tag: "1.18.0"
+  image_tag: "1.19.0"

Conclusion

In this demonstration we have seen how cepler can help you manage configuration files that define how a system should be deployed to multiple environments.

There are 3 basic commands in cepler check, prepare, record.

• cepler check -e <environment> - Check if an environment needs deploying
• cepler prepare -e <environment> - Prepare the state of the files checked out in the current directory for deployment
• cepler record -e <environment> - Record (and commit) metadata about files currently checked out and relevant to the environment

By using the cycle of:

$ cepler check -e <environment>
$ cepler prepare -e <environment>
$ <execute deploy command>
$ cepler record -e <environment>

We can ensure an orderly propagation of changes accross environments.

Here we have demonstrated this workflow using the cli commands on our local workstation. They are also particularly usefull when used within the context of a CI/CD system. Exploring cepler integration within a tool for workflow automation will be the subject of a future post.

You can use the help command to explore additional functionality and options:

% cepler help
cepler 0.4.5
USAGE:
    cepler [OPTIONS] <SUBCOMMAND>
FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information
OPTIONS:
        --clone <CLONE_DIR>                    Clone the repository into <dir>
    -c, --config <CONFIG_FILE>                 Cepler config file [env: CEPLER_CONF=]  [default: cepler.yml]
        --git-branch <GIT_BRANCH>              Branch for --clone option [env: GIT_BRANCH=]  [default: main]
        --git-private-key <GIT_PRIVATE_KEY>    Private key for --clone option [env: GIT_PRIVATE_KEY=]
        --git-url <GIT_URL>                    Remote url for --clone option [env: GIT_URL=]
SUBCOMMANDS:
    check        Check wether the environment needs deploying. Exit codes: 0 - needs deploying; 1 - internal error;
                 2 - nothing to deploy
    concourse    Subcommand for concourse integration
    help         Prints this message or the help of the given subcommand(s)
    ls           List all files relevent to a given environment
    prepare      Prepare workspace for hook execution
    record       Record the state of an environment in the statefile

The post Introducing Cepler appeared first on Stark & Wayne.

Habitat is an Open Source project that allows you to package your apps in a Platform/Runtime agnostic way.

This blog post discusses the recently added Cloud Foundry exporter that enables running Apps packaged in Habitat to run on Cloud Foundry.

Build the App

We'll be using this demo app as an example.

First lets build the app inside the habitat studio:

$ git clone https://github.com/cloudfoundry-community/expresso.git
$ cd expresso
$ hab studio enter
[1][default:/src:0]# build .
(...)
[2][default:/src:0]# ls results/*.hart
starkandwayne-expresso-0.1.0-20170927132309-x86_64-linux.hart

Exporting the App

Now that the app has been built, we face the issue of configuring it via the Cloud Foundry injected environment variables.
Habitat doesn't support configuring via arbitrary environment variables. Rather it relies on the hab-CLI being able to talk to an API to push configuration changes.

In a Cloud Foundry setting this won't work because the habitat http-API wont be exposed to the Cloud Foundry user.

Another way of configuring habitat based Apps is via a .toml file that overrides the default application configuration.
The Cloud Foundry exporter makes it possible to provide a mapping file that specifies how the environment variables should be rendered into the configuration .toml file.

[3][default:/src:0]# cat <<EOF >mapping.toml
> [app]
> port = "${PORT}"
> EOF
[4][default:/src:0]# hab pkg export cf starkandwayne/expresso ./mapping.toml
(...)
[4][default:/src:0]# exit

This will export 2 Docker images:

$ docker images
REPOSITORY                                  TAG                         IMAGE ID            CREATED             SIZE
starkandwayne/expresso                      cf-0.1.0-20170927132309     ed9e423b1ad8        11 seconds ago      170MB
starkandwayne/expresso                      0.1.0-20170927132309        05a01c4ecf99        22 seconds ago      167MB
starkandwayne/expresso                      latest                      05a01c4ecf99        22 seconds ago      167MB

The one with the cf- tag prefix is enabled to run on Cloud Foundry.

Running the App

In order to run the application you must be logged in with the cf-CLI to a Cloud Foundry instance that supports running Docker containers.

First we push the Docker container to dockerhub:

$ docker push starkandwayne/expresso:cf-0.1.0-20170927132309

Then we run it on Cloud Foundry:

$ cf push expresso -o starkandwayne/expresso:cf-0.1.0-20170927141758
(...)
$ cf app expresso
Showing health and status for app expresso in org test / space habitat as admin...
name:              expresso
requested state:   started
instances:         1/1
usage:             256M x 1 instances
routes:            expresso.local.pcfdev.io
last uploaded:     Sat 23 Sep 14:47:03 CEST 2017
stack:             cflinuxfs2
docker image:      starkandwayne/expresso:cf-0.1.0-20170927141758
     state     since                  cpu    memory          disk           details
#0   running   2017-09-23T12:48:04Z   0.2%   44.7M of 256M   556K of 512M

Once the image has been downloaded and the container started, we should be able access it via our browser or curl:

$ curl expresso.local.pcfdev.io
<!DOCTYPE html><html><head><title>Express</title><link rel="stylesheet" href="/stylesheets/style.css"></head><body><h1>Express</h1><p>Welcome to Express</p></body></html>%

Gotcha

In order for this to work you must make sure that the primary port is exposed and exported in the habitat plan.sh.

pkg_origin=starkandwayne
pkg_name=expresso
pkg_version=0.1.0
pkg_scaffolding=core/scaffolding-node
pkg_exports=(
  [port]=app.port
)
pkg_exposes=(port)

Other than adding the pkg_exports and pkg_exposes variables, the app is exactly the result of following the steps in this blog.

The post Running Habitat Apps on Cloud Foundry appeared first on Stark & Wayne.

In a recent blog post I briefly discussed how to build, export and run a service packaged via a Habitat plan.

In this post we will take a look at running Redis and backing it up via Shield.

Running Redis

To play around with the starkandwayne/redis release you can bring it up in the habitat studio:

$ hab studio enter
[1][default:/src:0]# hab svc load starkandwayne/redis
(...)
[2][default:/src:127]# hab pkg binlink starkandwayne/redis                                                                                                                   [4/1836]
» Symlinking redis-check-rdb from starkandwayne/redis into /bin
★ Binary redis-check-rdb from starkandwayne/redis/3.2.8/20170522110804 symlinked to /bin/redis-check-rdb
» Symlinking redis-server from starkandwayne/redis into /bin
★ Binary redis-server from starkandwayne/redis/3.2.8/20170522110804 symlinked to /bin/redis-server
(...)
[2][default:/src:0]# /bin/redis-cli -a password SET hello world
OK
[3][default:/src:0]# /bin/redis-cli -a password GET hello
"world"

Typing sl will give you the log output of the background supervisor that got started when you entered the studio:

[4][default:/src:0]# sl
--> Tailing the Habitat Supervisor's output (use 'Ctrl+c' to stop)
redis.default(O):  |    `-._`-._        _.-'_.-'    |
redis.default(O):   `-._    `-._`-.__.-'_.-'    _.-'
redis.default(O):       `-._    `-.__.-'    _.-'
redis.default(O):           `-._        _.-'
redis.default(O):               `-.__.-'
redis.default(O):
redis.default(O): 168:M 22 May 13:11:55.082 # Server started, Redis version 3.2.8
redis.default(O): 168:M 22 May 13:11:55.082 * The server is now ready to accept connections on port 6379
(...)

Running Shield daemon

Since Shield is a bit more complex system with a few moving parts I will run it via the pre-exported docker images in docker-compose.

First lets bring up the shield-daemon connected to a database. The daemon is the main coordinator of shield. It triggers backups as needed and persists the state of all created archives and backup jobs.

$ mkdir redis-hab-demo && cd redis-hab-demo                                                                                                                                      $ cat <<EOF > docker-compose.yml
version: '3'
services:
  shield:
    ports:
    - 443:443
    image: starkandwayne/shield
    command: "start starkandwayne/shield --peer database --bind database:postgresql.shield"
    links:
    - database
  database:
    image: starkandwayne/postgresql
    command: "start starkandwayne/postgresql --group shield"
EOF
docker-compose up

You can use the shield cli to interact with the daemon. Download it from the github-release.

From another terminal:

$ shield create-backend hab https://localhost
Successfully created backend 'hab', pointing to 'https://localhost'
Using https://localhost (hab) as SHIELD backend
$ export SHIELD_API_TOKEN=autoprovision

To actually backup a system you need to create a few entities in shield such as a policy, schedule and store. Lets create a schedule that takes a backup every day at 4am via the cli:

$ shield create-schedule -k
Schedule Name: daily
Summary:
Time Spec (i.e. 'daily 4am'): daily 4am
Schedule Name:                daily
Summary:
Time Spec (i.e. 'daily 4am'): daily 4am
Really create this schedule? [y/n] y
Created new schedule
Name:     daily
Summary:
Timespec: daily 4am
$ shield schedules -k
Name   Summary  Frequency / Interval (UTC)
====   =======  ==========================
daily           daily 4am

Because creating all entities manually is error prone we can also automate it by using the shield-agent.

Running Shield agent

The shield-agent is another component of Shield which is typically co-located with the data store you want to backup. You can configure it to automatically provision the elements that shield needs to run a backup.

Stop the docker-compose system via:

docker-compose stop && docker-compose rm -f

Use an EDITOR to add the agent to the docker-compose file. Add the agent service under the already existing services: key:

services:
  agent: # to autoprovision the dependant entities
    image: starkandwayne/shield-agent
    command: "start starkandwayne/shield-agent --bind daemon:shield.default --peer database"
    environment:
      HAB_SHIELD_AGENT: |
        [[stores]]
        name='local'
        plugin='fs'
        [stores.config]
        base_dir='/backups'
        [schedules]
        daily='daily 4am'
        [retention-policies]
        shortterm='86400'
    links:
    - database

Bring it up and lets see if it worked:

$ docker-compose up

Once everything is runnin you can see the configured entities in another terminal:

$ shield policies -k
Name       Summary  Expires in
====       =======  ==========
shortterm           1 days
$ shield stores -k
Name   Summary  Plugin  Configuration
====   =======  ======  =============
local           fs      {
                          "base_dir": "/backups"
                        }

Excellent we have now automatically configured a store. For the demo we are using the fs plugin to store backups in a local folder (/backups). In production you would want to use a plugin that can store the backups on a cloud based object store like s3.

Auto-configuring Redis

Now that we have a schedule, policy and store in place we can bring up Redis and have it automatically configure Shield to run backups.

Again stop the running system:

docker-compose stop && docker-compose rm -f

And add Redis to the docker-compose.yml. Again the redis service belongs under the already existing services: key. The volumes key new:

services:
  redis:
    image: starkandwayne/redis:edge
    volumes:
    - backups-volume:/backups
    ports:
    - 6379:6379
    command: "start starkandwayne/redis --peer shield --bind shield:shield.default"
    environment:
      HAB_REDIS: |
        bootstrap_from_backup=true
        backups_schedule='daily'
        backups_retention='shortterm'
        backups_store='local'
    links:
    - shield
volumes:
  backups-volume: {}

Bring it up and have a look:

$ docker-compose up

It can take a while for the whole system to come up but eventually you should see:

 % shield jobs -k
Name           P?  Summary  Retention Policy  Schedule  Remote IP        Target
====           ==  =======  ================  ========  =========        ======
redis-default  N            shortterm         daily     172.27.0.5:5444  {
                                                                           "base_dir": "/hab/svc/redis/data"
                                                                         }

So the Redis service we just added was able to configure its own backup job just by binding to a running Shield daemon. Cool!

Lets write a value, take a backup and see if it works:

$ redis-cli -a password SET hello world
OK
$ shield run redis-default -k
Scheduled immediate run of job
To view task, type shield task f82752ae-8066-4bca-9c71-47dc35464c80
$ shield archives -k
UUID                                  Target              Restore IP         Store         Taken at                         Expires at                       Status  Notes
====                                  ======              ==========         =====         ========                         ==========                       ======  =====
fb2b2b0b-925b-4e69-8083-ab649760048e  redis-default (fs)  192.168.16.5:5444  default (fs)  Tue, 16 May 2017 13:29:02 +0000  Wed, 17 May 2017 13:29:02 +0000  valid

So we set a value and manually took a backup. Lets destroy and recreate the Redis service. Thanks to the auto-bootstrapping feature the value should be restored without any further input:

$ docker-compose stop redis && docker-compose rm -f redis
Stopping hab_redis_1 ... done
Going to remove hab_redis_1
Removing hab_redis_1 ... done
$ docker-compose up -d redis
hab_database_1 is up-to-date
hab_agent_1 is up-to-date
hab_shield_1 is up-to-date
$ until redis-cli -a password GET hello; do echo 'Waiting for redis to bootstrap'; sleep 1; done
Waiting for redis to bootstrap
Waiting for redis to bootstrap
Waiting for redis to bootstrap
Waiting for redis to bootstrap
"world"

So thanks to Shield and Habitat's binding feature we are very easily able to add arbitrary Redis services all with backups preconfigured.

The post Use Shield to backup and restore Redis in Habitat appeared first on Stark & Wayne.

This is a hands on introduction to habitat.sh. It is a relatively new tool in the world of config management and aims to simplify many aspects of packaging / deploying and operating any kind of distributed systems regardless of the intended deploy target.

Some of the features that I have found particularly useful are:

• Simple / sane and reproducible approach to packaging.
• Supervision and handling lifecycles of processes.
• Service discovery including propagation of configuration to consuming services.

In this brief tour we will see some basic commands needed to build and run services packaged with habitat, without going into a lot of detail.

Setup

Lets start by cloning Stark & Wayne's habitat-plans repo.

$ git clone https://github.com/starkandwayne/habitat-plans.git
$ cd habitat-plans
$ ls
LICENSE         README.md       ci          mysql           postgresql      redis           scripts         shield          shield-agent    wordpress       wordpress-proxy

Here we can see the list of software that S&W engineers are working on to create 'Enterprise Ready' habitat plans.

We will use the hab cli to build and run these services so now would be a good time to install habitat.

$ hab --version
hab 0.21.0/20170421000356

Since we are building the plans under the starkandwayne origin we will need to generate an origin key for starkandwayne. Every package in Habitat belongs to an origin, and is cryptographically signed with that origin's private key.

$ hab origin key generate starkandwayne
» Generating origin key for starkandwayne
★ Generated origin key pair starkandwayne-20170424121434.

Since I will be demonstrating the workflow in a Mac OS environment you will also need Docker for Mac installed.

Building a package

Once your environment is set up you can enter the habitat studio (a shell environment setup to quickly iterate on your habitat plans). On a Mac this will download and start a docker container. On Linux this will simply chroot

$ HAB_ORIGIN=starkandwayne hab studio enter
(...)
[0][default:/src:0]#

A habitat package is defined via a plan.sh file. Lets take a look at the one for postgresql.

[0][default:/src:0]# ls postgresql
config  default.toml  hooks  plan.sh  tests
[1][default:/src:0]# less postgresql/plan.sh
(...)

You will see a bunch of variables:

pkg_name=postgresql
pkg_version=9.6.1
pkg_origin=starkandwayne
pkg_description="PostgreSQL is a powerful, open source object-relational database system."
pkg_upstream_url="https://www.postgresql.org/"
pkg_license=('PostgreSQL')
pkg_source=https://ftp.postgresql.org/pub/source/v${pkg_version}/${pkg_name}-${pkg_version}.tar.bz2
(...)

And further down some hooks that start with do_*

do_build() {
        # ld manpage: "If -rpath is not used when linking an ELF
        # executable, the contents of the environment variable LD_RUN_PATH
        # will be used if it is defined"
        ./configure --disable-rpath \
              --with-openssl \
              --prefix="$pkg_prefix" \
              --with-uuid=ossp \
              --with-includes="$LD_INCLUDE_PATH" \
              --with-libraries="$LD_LIBRARY_PATH" \
              --sysconfdir="$pkg_svc_config_path" \
              --localstatedir="$pkg_svc_var_path"
        make world
}
do_install() {
  make install-world
}

These functions and variables instruct habitat on how to create the starkandwayne/postgresql package. Let's build it now:

[2][default:/src:0]# build postgresql

This will take a while. The Postgresql source is downloaded and compiled according to the plan.sh. Eventually, you will see in the output that the build has completed:

(...)
   postgresql: hab-plan-build cleanup
   postgresql:
   postgresql: Source Path: /hab/cache/src/postgresql-9.6.1
   postgresql: Installed Path: /hab/pkgs/starkandwayne/postgresql/9.6.1/20170424121553
   postgresql: Artifact: /src/results/starkandwayne-postgresql-9.6.1-20170424121553-x86_64-linux.hart
   postgresql: Build Report: /src/results/last_build.env
   postgresql: SHA256 Checksum: 6187c711bd1f4471722053754b068f8f6de048612ca2b9a00aaec08f2ce6eb17
   postgresql: Blake2b Checksum: ae297ffaa007478eea6fba9676c2170e0af6dbb025dfdb61cf9512178a636fb2
   postgresql:
   postgresql: I love it when a plan.sh comes together.
   postgresql:
   postgresql: Build time: 5m23s

The resulting *.hart file (which is the resulting artifact) will reside under the results/ directory. This directory will also be present on your host machine because the directory you were in got mounted under /src when you typed hab studio enter.

[3][default:/src:0]# ls results
last_build.env  starkandwayne-postgresql-9.6.1-20170424121553-x86_64-linux.hart

Habitat has also already installed your built plan into the studio under /hab/pkgs/...

[4][default:/src:0]# find /hab/pkgs/starkandwayne -type d -maxdepth 4
/hab/pkgs/starkandwayne
/hab/pkgs/starkandwayne/postgresql
/hab/pkgs/starkandwayne/postgresql/9.6.1
/hab/pkgs/starkandwayne/postgresql/9.6.1/20170424121553
/hab/pkgs/starkandwayne/postgresql/9.6.1/20170424121553/share
/hab/pkgs/starkandwayne/postgresql/9.6.1/20170424121553/lib
/hab/pkgs/starkandwayne/postgresql/9.6.1/20170424121553/config
/hab/pkgs/starkandwayne/postgresql/9.6.1/20170424121553/bin
/hab/pkgs/starkandwayne/postgresql/9.6.1/20170424121553/include
/hab/pkgs/starkandwayne/postgresql/9.6.1/20170424121553/hooks

Running your built service

To start your built package in the habitat studio you can simply run:

[5][default:/src:0]# hab start starkandwayne/postgresql
hab-sup(MR): Butterfly Member ID 67642576ef154e76b8160c83c605b475
hab-sup(SR): Adding starkandwayne/postgresql/9.6.1/20170424122104
hab-sup(MR): Starting butterfly on 0.0.0.0:9638
hab-sup(MR): Starting http-gateway on 0.0.0.0:9631
postgresql.default(SR): Initializing
postgresql.default(SV): Starting process as user=hab, group=hab
postgresql.default(O): Starting PostgreSQL
postgresql.default(O): 2017-04-24 12:45:35.587 GMT: FATAL:  could not open shared memory segment "/PostgreSQL.1733059837": Permission denied
postgresql.default(O): 2017-04-24 12:45:35.597 GMT: LOG:  database system is shut down
hab-sup(SV): postgresql.default - process 15914 died with exit code 1
hab-sup(SV): postgresql.default - Service exited

Due to the constraints of the studio context this package actually won't start the way we want it (Permission denied). But that doesn't really matter! Since I am introducing you to a possible mac workflow you will need to export your packages to docker anyway.

[6][default:/src:0]#  hab pkg export docker starkandwayne/postgresql
(...)

Once we have exported the package we can exit the studio and run it in docker on our host machine:

[7][default:/src:0]# exit
$ docker images
REPOSITORY                                  TAG                    IMAGE ID            CREATED              SIZE
starkandwayne/postgresql                    9.6.1-20170424122104   938e89b8286b        About a minute ago   429 MB
starkandwayne/postgresql                    latest                 938e89b8286b        About a minute ago   429 MB
$
$ docker run -p 5432:5432 starkandwayne/postgresql
hab-sup(MR): Butterfly Member ID eb68d2c1489a4c8084ac824311702750
hab-sup(SR): Adding starkandwayne/postgresql/9.6.1/20170424122104
hab-sup(MR): Starting butterfly on 0.0.0.0:9638
hab-sup(MR): Starting http-gateway on 0.0.0.0:9631
hab-sup(SC): Updated pwfile 30087c89f3dfc2b2179a2b567a67f094cf1cd8b1c6e3cc1473aef95fe65de533
hab-sup(SC): Updated postgresql.conf 3ad4f841f942d1ae606ad89a54cb0ebe8a4313a1af4f1a8958c50b3aedaeb6ef
hab-sup(SC): Updated pg_hba.conf cc1489fff1c05e7b547fac8018dbe22d0327a8ea6a288a6604a1b6c8232619e1
postgresql.default(SR): Initializing
(...)

You can test connecting to postgres from another terminal:

$ cat postgresql/default.toml | egrep '(name|password)'
name = 'admin'
password = 'admin'
$ PGPASSWORD=admin psql -h localhost -U admin postgres
psql (9.6.2, server 9.6.1)
Type "help" for help.
postgres=# \q

Conclusion

This was a very quick introduction of how to build, export and run a PostgreSQL service packaged via habitat.

To learn more about habitat please go through the tutorial on the habitat.sh website. It does a good job of explaining the basic concepts while walking you through the creation of your own plan.
The docs are also an excellent point of reference.

Also since Stark & Wayne have begun contributing plans to the habitat community you can keep following our blog for more habitat related content including workflow tips / best practices and other topics.

The post Mini Tour of Habitat appeared first on Stark & Wayne.

Sometimes when helping our clients get their Cloud Foundry installation in line with what we consider best practices we determine the best strategy to achieve this is to create a brand new deployment and migrate all existing workload.

In this blog post I will describe the steps we took to perform a migration from an old v245 CF installation with DEA and Diego cells, using PG as a database to a new v247 installation using MySQL db with only diego cells as runners.

EDIT:
Though we love PostgreSQL this migration was necessary due to customer requirements. In particular the HA properties of the MySQL BOSH release are well tested and widely used in many prod Cloud Foundry installations. Adapting a PostgreSQL release to support the required HA properties would have taken more time than we had to complete the project.

This kind of migration has a number of gotchas and it took us many days of trial and error at each step of the way before we had a plan that consistently gave us the desired result. That being said this is just one solution that we found to work in our exact situation given the exact constraints we faced. I'm sure there are more elegant solutions and steps that will have to be done differently in other situations.

Preparation

This migration took place on AWS. The new installation was deployed into the same account and region as the old one. A vpc_peering_connection was created via terraform and connected to the routing tables of the old and new VPC so that network connectivity was present.

1) CF Properties

When deploying the new CF we configured the UAA to pre-create a user with identical credentials as the existing admin user.
This made sense so that smoke-tests (which were part of the genesis managed CI pipeline) could be run in the new environment regardless of where the DNS record for the api was pointing.
Also the setting under properties.cc.db_encryption_key had to be identical for the cloud_controller to be able to read the database after migration.

2) Migration tools

To do the actual migration from Pg -> MySQL we used a proprietary tool (50$) that can be found here. Out of all the methods we tried this was the most reliable! It comes as an exe and will need to be installed on a Windows box that has network access to both the Pg and MySQL instance. We deployed the box into the AWS vpc of the old installation and made sure the security_groups allowed access to pg and the leading mysql_proxy. We used Remote Desktop to access the box. We also installed MySQL Workbench to truncate the databases but this could also be done from the command line.

3) Smoke Tests

Before beginning the migration we pointed the DNS entry to the new environment just to run the cf smoke tests 1 time. This caused ~15minutes of downtime. Once we had validated the new environment we pointed the DNS back.

Migration

Here is how we actually migrated the data.

1) Stop jobs

Use BOSH to stop all jobs associated with the databases prior to migration.
These are: api_z1, api_z2, uaa_z1 and uaa_z2 jobs

2) Backup

We took a shield-backup from the new environment (uaadb + ccdb). If things go wrong when migrating its always nice to be able to jump back to a known good point.

3) Truncating tables

Before migrating the data be sure that all the tables (of the new environment :-p) are completely empty!
With 1 exception:
IMPORTANT!!! Be sure not to truncate or alter the schema_migrations table in either database. This table should stay untouched by the migration of the rest of the data.

For the uaadb truncation could be achieved with the TRUNCATE TABLE sql for each table.
For ccdb this didn't work as nicely because some tables have foreign key constraints and cannot be truncated. For the tables of the ccdb where TRUNCATE TABLE doesn't work the individual rows need to be removed via DELETE FROM sql.

4) Backup

Take another backup when all tables (except schema_migrations) are empty. This is also a useful point to jump back to when things don't work.

5) Data migration

Start the exe of the installed migration tool this will start a walkthrough that guides through the migration process. Enter the details of the connection, select the database name (we started with uaadb) and schema (public). At one point you can select the migration strategy. There are options regarding wether tables should be dropped and the schema should be migrated or not. Out of all options the only one that worked consistently was choosing merge. This leaves the existing tables as they are and merely transfers the rows. The last step before the data is actually transferred is selecting which tables to migrate. Select them all except schema_migrations.

6) One more backup

Once the migration is done, you may as well back this up as well.

7) Migrate blobstores

As a last step before starting the jobs back up the blobs stored in the cf blobstore must be migrated. Since we were using S3 as a blobstore for both the old and new installation we were able to perform this step using the aws s3 sync ... command.

Running the new CF v247

At this point all the data to start all the existing apps on the new CF had been migrated. Moving forward from here is committing to running the apps on the new platform. Trying these steps out multiple times on a staging environment is highly recommended.

1) Restart the jobs

With bosh start all all the previously stopped jobs will start back up.
This should cause the apps to be scheduled on the diego runtime.

2) Switch DNS

Once the apps start up pointing the DNS record to the new API will make them accessible via the new platform.
At this point rerunning the smoke-tests will ensure that things are working as expected.

3) Services

Since the scope of the migration only applied to the core platform all the existing backing services needed to be made accessible from the new deployment so that apps utilizing them could continue to function properly. For this most of them needed to be redeployed with knowledge of the new nats endpoint and credentials so that they could advertise their endpoint to CF.

4) Enabling diego

Since we still had some old style runners in the old deployment not all the apps would start up automatically. Using the Diego-Enabler cf-cli plugin it was easy to identify and enabling all apps to run on diego which caused them to automatically be scheduled and work

Celebrate

Once all that has gone well a round of congratulations is in order!

If this blogpost was useful to you or you have further questions let me know in the comments.

The post Migrating Cloud Foundry from PG to MySQL: A Report appeared first on Stark & Wayne.

As a side project I have started looking at mesos and the mesosphere ecosystem. This blog post documents the first steps to take to get a mesos deployment running locally via bosh-lite.

What is mesos?

From the mesos website:

Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.

In practice it is a platform for scheduling workloads across a number of nodes that offer compute resources.

As a platform it doesn't make scheduling decisions itself. Instead it allows registering frameworks that get offered resources which then make the scheduling decisions for the particular kinds of workloads the framework is designed to manage.

You can find an architectural overview here.

Running mesos locally

The easiest way to get a complete mesos cluster running locally is by deploying it onto bosh-lite.
cloudfoundry-community/mesos-boshrelease has up to date versions (as of Sep. 2016) of mesos, zookeeper and marathon (a framework for orchastrating containers on mesos).

Make sure your bosh-lite is running and you have uploaded a recent stemcell.

git clone https://github.com/cloudfoundry-community/mesos-boshrelease
cd mesos-boshrelease
bosh target 192.168.50.4:25555
bosh upload release releases/mesos/mesos-6.yml
bosh update cloud-config templates/warden-cloud.yml
bosh deployment templates/deployment.yml
bosh status --uuid | pbcopy
vi templates/deployment.yml

before deploying you must paste the director-uuid into the deployment.yml file.

director_uuid: <%= '<director-uuid>' %>

then continue with the deployment

bosh -n deploy

This is a good time to have a coffee break. During the deployment process mesos will be compiled which can take a long time! >25minutes on my MBP 3.1Ghz 16GB.

...

Welcome back! Hope the deployment went smoothly!

% bosh vms
Acting as user 'admin' on 'Bosh Lite Director'
Deployment 'mesos-deployment'
Director task 563
Task 563 done
+-------------------------------------------------------+---------+----+---------+-------------+
| VM                                                    | State   | AZ | VM Type | IPs         |
+-------------------------------------------------------+---------+----+---------+-------------+
| marathon/0 (5b25e15f-5d25-4c37-b887-c0036ca0a8b4)     | running | z1 | medium  | 10.244.10.5 |
| marathon/1 (d50dc3a5-6572-4a42-9868-9bf25ee3d2bd)     | running | z2 | medium  | 10.244.11.5 |
| marathon/2 (86262046-f74e-4cca-a24f-1bd74a36f633)     | running | z3 | medium  | 10.244.12.5 |
| mesos-agent/0 (1bdf8491-7504-4ab1-8084-908bd10b950d)  | running | z1 | medium  | 10.244.10.4 |
| mesos-agent/1 (413a3989-570e-4fd3-8bf4-9f73a953bca8)  | running | z2 | medium  | 10.244.11.4 |
| mesos-agent/2 (dc1b927a-49eb-4a7b-bfab-3545f30f9aa3)  | running | z3 | medium  | 10.244.12.4 |
| mesos-master/0 (366ddf6f-0e91-444b-bc0c-41384461929e) | running | z1 | medium  | 10.244.10.3 |
| mesos-master/1 (55b89a73-8645-45e0-bce8-d1e869d79cab) | running | z2 | medium  | 10.244.11.3 |
| mesos-master/2 (a6da5a7e-7cb7-4797-9b32-754e01620cc2) | running | z3 | medium  | 10.244.12.3 |
| zookeeper/0 (afb7ae65-33bd-4c43-9b5e-2787946b2066)    | running | z1 | medium  | 10.244.10.2 |
| zookeeper/1 (485d6645-b521-4add-9d53-51801a7e1061)    | running | z2 | medium  | 10.244.11.2 |
| zookeeper/2 (e3c34158-af90-40a0-a12a-4eadf0fa70db)    | running | z3 | medium  | 10.244.12.2 |
+-------------------------------------------------------+---------+----+---------+-------------+
VMs total: 12

Looks like we have:

• 3 zookeeper vms. These are used as an HA key-value store for leader election and persisting state.
• 3 mesos-master vms for HA.
• 3 mesos-agent vms for running workloads.
• 3 marathon vms for HA orchestrating of container workloads.

Interacting with mesos

In order to access the network that the bosh-lite garden containers are on be sure to run

bin/add-route
``` from the root of the bosh-lite repo before proceeding.
We can curl any of the mesos-master nodes to get information about the running cluster.

% curl -L 10.244.10.3:5050/state | jq
(omitted)

We use `-L` to follow redirects to the current leader.

open http://10.244.10.3:5050

Will show us the default mesos ui. (Due to [a bug](https://issues.apache.org/jira/browse/MESOS-5911) redirection to the leader will not work in a browser so you may have to visit `10.244.11.3:5050` or `10.244.12.3:5050` instead.)
Clicking on [agents](http://10.244.10.3:5050/#/agents) will show a list of registered agents.
Clicking on [frameworks](http://10.244.10.3:5050/#/frameworks) will show you that 1 framework (marathon) is registered.
You can find the [web-ui](http://10.244.11.5:8080/ui/#/apps) for marathon by clicking on the framework id and then the link next to `Web UI`. Username and password for http basic-auth when accessing the marathon ui are `marathon` and `marathon`.
## Deploying a container via Marathon
To wrap up this introduction to mesos we will run an actual workload via marathon. For this we will deploy an alternative web-ui for mesos.

git clone https://github.com/Capgemini/mesos-ui
cd mesos-ui
vi marathon.json

Edit the `ZOOKEEPER_ADDRESS` entry in `marathon.json` to point to our local deployment:

"ZOOKEEPER_ADDRESS": "10.244.10.2:2181,10.244.11.2:2181,10.244.12.2:2181"

Then:

curl -X POST -HContent-Type:application/json -d @marathon.json marathon:marathon@10.244.10.5:8080/v2/apps

to have marathon deploy the container.
Go back to the marathon ui to watch the deployment process (can take a few minutes due to the docker image being downloaded).
Once its deployed you can find the newly deployed mesos-ui by clicking through the marathon ui: applications -> mesos-ui -> instance-id -> endpoints
Congratulations you have run your first workload on a locally running mesos-cluster.

The post Investigating mesos appeared first on Stark & Wayne.

At Stark & Wayne we are concerned with the complete lifecycle of deployments and simple upgrade paths are an important ingredient to any long running system.

For users of the cf-containers-broker (a service broker that will let you run and bind services that live in docker containers) upgrading their services just got much simpler.
By running bin/update_all_containers all containers will be restarted with the latest version of the image and configuration specified in the service plans.

When consuming the cf-containers-broker via the cloudfoundry-community/docker-boshrelease there is now a job property that will trigger automatic updates to all containers everytime the job is redeployed. This feature is available as of v25 of the BOSH release.

How does it work?

Because the cf-containers-broker itself is stateless it uses the custom labels feature of Docker to add the plan_id and instance_id of the service as metadata when creating the container.

This allows it to map running containers to service plans for upating.

Caveats

Since this feature relies on the existence of labels on the container it will only work with services that have been started with the latest version of the cf-containers-broker.

Containers that were started with an older version of the broker will not be affected and continue running in the state they were originally configured.

To allow the broker to update pre-existing service instances run:

cf update-service <service-id>

This will allow the broker to add the necessary labels while recreating the service.

This command will only work if the service is advertized in the catalog as planupdateable: true.

The post Keeping Dockerized services up to date appeared first on Stark & Wayne.