Stark & Wayne
  • by Justin Carter

Introduction

Deploying software to multiple environments (such as dev / staging / production) introduces operational complexity that requires explicit managing in order to ensure parity between environments.

Previously I wrote an article introducing how you can use cepler to significantly reduce this overhead (and it is recommended to read that first).

In this article we will use cepler-templates to automate the execution of the cepler check, cepler prepare, cepler record cycle to deploy cf-for-k8s using GitHub actions.

Prerequisites

If you want to follow along with the demo, you will need access to two publicly accessible Kubernetes clusters representing two environments that we want to deploy Cloud Foundry to.

If you want the resulting Cloud Foundry to be fully functional you will also need a DNS name you can use to access Cloud Foundry.

Preparation

First, we will clone the cf-for-k8s repo and generate some values we need.

$ git clone https://github.com/cloudfoundry/cf-for-k8s && cd cf-for-k8s
$ git checkout v1.0.0
$ ./hack/generate-values.sh -d <testflight-dns> > testflight-values.yml
$ ./hack/generate-values.sh -d <staging-dns> > staging-values.yml

Then, we will create a new repository to store the files needed to deploy cf-for-k8s.

$ cd ..
$ git clone git@github.com:your-github-org/cf-k8s-cepler.git && cd cf-k8s-cepler

Config Files

We will use vendir to sync the files we need from cf-for-k8s:

$ mkdir k8s
$ cat <<EOF > vendir.yml
---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
minimumRequiredVersion: 0.8.0
directories:
- path: k8s
  contents:
  - path: cf-for-k8s
    git:
      url: https://github.com/cloudfoundry/cf-for-k8s
      ref: v1.0.0
    includePaths:
    - config/**/*
EOF
$ vendir sync
$ git add . && git commit -m 'Sync cf-for-k8s config files'

Now, we need to add the values we generated in the previous step and append some Docker Hub credentials to it. Don't actually add your password here, that will be injected via the GitHub secrets mechanism.

$ cp ../cf-for-k8s/testflight-values.yml ./k8s/
cat <<EOF >> k8s/testflight-values.yml
app_registry:
  hostname: https://index.docker.io/v1/
  repository_prefix: "<dockerhub_username>"
  username: "<dockerhub_username>"
  password: DUMMY
EOF
$ cp ../cf-for-k8s/staging-values.yml ./k8s/
cat <<EOF >> k8s/staging-values.yml
app_registry:
  hostname: https://index.docker.io/v1/
  repository_prefix: "<dockerhub_username>"
  username: "<dockerhub_username>"
  password: DUMMY
EOF
$ git add . && git commit -m 'Add environment-values'

Next, we will add a cepler.yml and ci.yml which are needed to generate the deployment pipeline.

$ cat <<EOF > cepler.yml
environments:
  testflight:
    latest:
    - k8s/cf-for-k8s/**/*
    - k8s/testflight-values.yml
  staging:
    passed: testflight
    propagated:
    - k8s/cf-for-k8s/**/*
    latest:
    - k8s/staging-values.yml
EOF
$ cat <<EOF > ci.yml
cepler:
  config: cepler.yml

driver:
  type: github
  repo:
    access_token: ${{ secrets.ACCESS_TOKEN }}
    branch: master
  secrets:
    app_registry:
      password: ${{ secrets.DOCKERHUB_PASSWORD }}

processor:
  type: ytt
  files:
  - k8s/cf-for-k8s/config
  - k8s/*.yml

executor:
  type: kapp
  environments:
    testflight:
      app_name: testflight-cf
      ca_cert: ${{ secrets.TESTFLIGHT_CA }}
      server: ${{ secrets.TESTFLIGHT_SERVER }}
      token: ${{ secrets.TESTFLIGHT_TOKEN }}
    staging:
      app_name: staging-cf
      ca_cert: ${{ secrets.STAGING_CA }}
      server: ${{ secrets.STAGING_SERVER }}
      token: ${{ secrets.STAGING_TOKEN }}
EOF
$ git add . && git commit -m 'Add cepler.yml and ci.yml'

The cepler.yml file configures the order in which the environments are deployed and which files belong to each environment.

The ci.yml tells the cepler-templates processor how the CD pipeline should be built. In this case, we are using github as a driver ytt as a processor and kapp as an executor.

Secrets

Now, go to https://github.com/your-github-org/cf-k8s-cepler/settings/secrets and add the following secrets:

To gain access, we also need a token from a service account that has the required permissions for deploying cf:

$ kubectl config set-context <cluster>
$ kubectl apply -f https://raw.githubusercontent.com/starkandwayne/cf-k8s-cepler/master/deployer-account.yml
$ secret_name=$(kubectl get serviceaccount cf-deployer -o json | jq -r '.secrets[0].name')
$ kubectl get secrets ${secret_name} -o json | jq -r '.data.token' | base64 --decode
<token>

Copy the resulting token into the TESTFLIGHT_TOKEN and STAGING_TOKEN secrets respectively.

Configuring github-actions

Now everything is in place and we can create our continuous deployment setup:

$ workflows_dir=.github/workflows
$ mkdir -p ${workflows_dir}
$ docker run -v $(pwd):/workspace/inputs -it bodymindarts/cepler-templates:0.2.0 > ${workflows_dir}/deploy-cf-environments.yml
$ git add . && git commit -m 'Add deploy-cf-environments workflow'

Deployment

Once you have pushed your repo upstream to GitHub, an action should be kicked off:

$ git push -u origin master

You can follow the progress of the triggered workflows here: https://github.com/your-github-org/cf-k8s-cepler/actions

If you click on the latest commit you will be able to drop down the cepler-deploy workflow.

There are two jobs; deploy-testflight and deploy-staging.

The initial run of deploy-staging should fail since it depends on testflight having been completed successfully at least once. If you follow the deploy-testflight job the deployment should complete correctly and produce an updated cepler state:

$ git pull
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 1), reused 4 (delta 1), pack-reused 0
Unpacking objects: 100% (4/4), 5.83 KiB | 2.92 MiB/s, done.
From github.com:bodymindarts/cf-k8s-cepler
   b68074a..61d0273  master     -> origin/master
Updating b68074a..61d0273
Fast-forward
 .cepler/testflight.state | 566 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 566 insertions(+)
 create mode 100644 .cepler/testflight.state

This commit should in-turn trigger another run of the workflow. You can go back to https://github.com/your-github-org/cf-k8s-cepler/actions and click on the latest commit [cepler] Updated testflight state to watch the next deploy.
This time the deploy-testflight job should complete as a no-op. The deploy-staging job should complete successfully creating another commit.

From here on, any change to the files referenced in the cepler.yml file should trigger successive deploys. Hence, we have achieved a continuous deployment pipeline that deploys cf-for-k8s to successive environments via GitHub actions.

Accessing CF

To check that things are working as expected, you can find out the external IP of the istio-ingressgateway and point your DNS entry to it:

$ kubectl get svc -n istio-system
NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                                                      AGE
istio-ingressgateway   LoadBalancer   192.168.77.107   35.246.241.158   15021:31048/TCP,80:30241/TCP,443:30963/TCP,15443:30925/TCP   23m
istiod                 ClusterIP      192.168.90.225   <none>           15010/TCP,15012/TCP,443/TCP,15014/TCP,853/TCP                23m

Then once your DNS has been updated run:

$ cf api api.<testflight-dns> --skip-ssl-validation
$ cf auth admin `cat k8s/testflight-values.yml | yq -r .cf_admin_password`

Conclusion

We have demonstrated how using cepler-templates can make it very simple to pipeline a complex system like cf-for-k8s using the ci.yml and cepler.yml as configuration inputs.