How to rotate Kubernetes secrets with Quarks and KubeCF?

One of the brilliant aspects of BOSH that has been brought across to Kubernetes by the Quarks & KubeCF teams has been the generation of internal secrets. Internal client needs a secret to talk to internal Redis? No one cares what it is; just generate a good one and share it with the two entities. Need x509 certificates? Great, generate and share those too using a self-signed root CA, which was also internally generated. Brilliant stuff.

But after a Secret is generated initially, how do I ask Quarks to regenerate it? I know how to create a daily cronjob in Kubernetes – a fabulous feature of Kubernetes is a builtin cronjob facility – but I don’t know what to do in my cronjob to rotate the Quarks-managed Secrets.

What is Quarks? What is KubeCF? What is cf-operator?

Quarks is a project name within the Cloud Foundry Foundation for a collection of projects that are helping to phase in the migration of non-Kubernetes Cloud Foundry into Cloud Foundry as-a-first-class-citizen of Kubernetes over time. During 2019, the Cloud Foundry code bases were only packaged for BOSH. The Quarks initiative was to consume these BOSH releases – with their scripts for package compilation, configuration files, and runtime bindings – and produce artifacts that could be run atop any Kubernetes.

The resulting deployable project is called KubeCF, previously named SCF v3. It is a Helm chart. It installs resources into Kubernetes, and eventually you have a running Cloud Foundry that uses Kubernetes itself to run your applications (thanks to the Eirini project).

If you are looking for a commercial distribution of KubeCF, then please get in contact with the great folks at SUSE and ask for SUSE Cloud Application Platform. Also, myself and several others from Stark & Wayne will be at SUSECON 2020 in Dublin in March.

Does the KubeCF Helm chart install Kubernetes Deployments and Services? No. The Quarks initiative adds an invisible (to you the human operator) step, thanks to cf-operator.

The cf-operator project is the hardcode power behind Quarks initiative. It converts BOSH releases, BOSH manifests, and BOSH operator files, such as for the upstream Cloud Foundry BOSH releases, into running Kubernetes pods.

The KubeCF Helm chart actually installs a Kubernetes Custom Resource called BOSHDeployment, and the cf-operator watches for it, and reconciles it eventually into a set of pods that are Cloud Foundry.

In addition to pods, cf-operator converts the BOSHDeployment resource into dozens of secrets, such as internal passwords, and certificates.

Generating secrets

To generate a password with cf-operator, create a QuarksSecret. The cf-operator will reconcile it into a shiny new Kubernetes Secret that you can use in any project – Cloud Foundry, or vanilla Kubernetes. It’s very cool.

# generate-secret.yamlapiVersion: quarks.cloudfoundry.org/v1alpha1
kind: QuarksSecret
metadata:
  name: my-internal-secret
spec:
  type: password
  secretName: my-internal-secret

Install this resource into the same namespace as cf-operator (probably kubecf if you’ve already got KubeCF installed).

$ kubectl apply -f generate-secret.yaml -n kubecf
$ kubectl get quarkssecret my-internal-secret
NAME                 AGE
my-internal-secret   24s

cf-operator will reconcile this QuarksSecret into a new Secret secretName: my-internal-secret:

$ kubectl get secret my-internal-secret -n kubecf
NAME                 TYPE     DATA   AGE
my-internal-secret   Opaque   1      66s
$ kubectl get secret my-internal-secret -n kubecf -ojsonpath='{.data.password}' | base64 --decode
Q3ba8rLvBdyzcqFmArAUTTxzuerGqfVGBBMl6cyWhH7AFgjS3Ys3wl4eIkimEljA

Awesome. Now any pod could attach this secret and the password will be available in a volume or environment variable.

Requirement for rotating secrets

There are two requirements for rotating secrets:

• Ability to update a Secret with new values
• Entities must watch for changes in the Secret and start using the new values

The latter requirement means we cannot use environment variables to access secrets. If the Secret changes, the Pods will not be restarted to get the new environment variables.

Instead, to support secret rotation we must expose Secrets into Pods using volumes. When the Secret changes, the new values are automatically populated into the files of that volume.

The applications/processes running within the Pod’s containers must also be written to observe changes to Secrets in volume files.

Alternately, if your applications/processes can not watch for changes to Secrets in volume files, then the Pods must fail hard and have Pods be recreated (with the new secrets passed to new processes).

Rotating secrets with Quarks

Quarks and cf-operator allowed us to generate a Secret using a QuarksSecret custom resource. Fortunately, it also allows us to rotate or regenerate the Secret’s contents.

# rotate-my-internal-secret.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rotate-my-internal-secret
  labels:
    quarks.cloudfoundry.org/secret-rotation: "true"
data:
  secrets: '["my-internal-secret"]'

To install our request to rotate my-internal-secret Secret:

kubectl apply -f rotate-my-internal-secret.yaml

This creates a (short-lived) ConfigMap:

$ kubectl get cm rotate-my-internal-secret
NAME                        DATA   AGE
rotate-my-internal-secret   1      40s

The ConfigMap acts as a trigger to cf-operator to regenerate one or more Secrets from their QuarksSecret definition.

We can see that cf-operator has changed the .data.password value of our Secret:

$ kubectl get secret my-internal-secret -n kubecf -ojsonpath='{.data.password}' | base64 --decode
X1aidUZ2MBdtkYJccNdR4xWyJr6JcUtvh4LBafGsL38qpkPnX1kSBhR3sHCQRmiJ

We can now delete the ConfigMap trigger resource:

kubectl delete -f rotate-my-internal-secret.yaml

Want to generate the Secret again? Re-install the trigger ConfigMap:

$ kubectl apply -f rotate-my-internal-secret.yaml
$ kubectl get secret my-internal-secret -n kubecf -ojsonpath='{.data.password}' | base64 --decode
79XERALdMRQi0LxonEoTJuV8o8ZUGmgUjjHqRp5yAVY8bnCciBoJFEdFDBoI7Du1

Does KubeCF support secret rotation?

At the time of writing this article I’ve not yet tested all the different Secrets in KubeCF to confirm that all clients and servers support secret rotation. Stay tuned.