Mental Migration: A Cloud Foundry Operator’s Perspective on Starting Kubernetes – Part 2

This is the second part of my journey as a Cloud Foundry operator learning about the various components Kubernetes by comparing the components to functionalities I know in Cloud Foundry. The start of this series is here: https://starkandwayne.com/blog/mental-migration-a-cf-operators-perspective-on-starting-kubernetes/

Source of Truth – Cloud Foundry

In Cloud Foundry land there are relational databases that contain the desired state of the platform. If you have your PostgreSQL/MySQL databases for the Cloud Controller, UAA, Diego, Silk, and Locket as externally deployed resources, you can completely delete all the Cloud Foundry vms and deployment, count to 100, redeploy CF via BOSH. Other than the Change Control Board representative standing at your desk giving you dirty looks (Hi Jean!), CF will come back up happily and continue to serve up apps.

If you start to dig into the Databases of Cloud Foundry, especially the Cloud Controller Database (ccdb), the desired state of the platform is represented across a few tables. There are instructions here for connecting directly to the database. Information on the desired state of apps, orgs, spaces, processes, quotas, and services are stored here. There are a few interesting queries documented at here and here which can help you gain some insights on your CF deployment.

Below is a list of the tables in the ccdb to give you a rough idea of the objects which are tracked:

app_eventssecurity_groups
app_usage_events
apps
apps_routes
apps_v3
billing_events
buildpack_lifecycle_data
buildpacks
delayed_jobs
domains
droplets
env_groups
events
feature_flags
isolation_segments
lockings
organizations
organizations_auditors
organizations_billing_managers
organizations_managers
organizations_private_domains
organizations_users
package_docker_data
packages
quota_definitions
route_bindings
route_mappings
routes
schema_migrations
security_groups_spaces
service_auth_tokens
service_bindings
service_brokers
service_dashboard_clients
service_instance_operations
service_instances
service_keys
service_plan_visibilities
service_plans
service_usage_events
services
snw_delayed_jobs_backup
space_quota_definitions
spaces
spaces_auditors
spaces_developers
spaces_managers
stacks
tasks
users
v3_droplets
v3_service_bindings

If you have been using Cloud Foundry long enough you’ll remember a time where etcd was used to store “current state”. Upgrades at times became “challenging” because of the number of times etcd would simply refuse to come up clean, this blog post was used several times to glue large deployments back together, typically after 3 am. In short, you would wipe the etcd data store and let CF repopulate the data.

Source of Truth – Kubernetes

Instead of a relational database like PostgreSQL or MySQL, Kubernetes uses etcd to store information about the clusters. Desired and current state is stored in etcd. Information such as secrets, resources, pods, deployments, services and networking definitions are stored in the etcd data store.

Like Cloud Foundry, Kubernetes uses an API layer to abstract direct access to the etcd data store. Accessing the data indirectly is done with the cf and kubectl CLIs.

Below is a summary of the keys on a minikube Kubernetes deployment:

/registry/apiregistration.k8s.io/apiservices/v1/*
/registry/clusterrolebindings/cluster-admin
/registry/clusterrolebindings/kubeadm:*
/registry/clusterrolebindings/minikube-rbac
/registry/clusterrolebindings/storage-provisioner
/registry/clusterrolebindings/system:*
/registry/clusterroles/admin
/registry/clusterroles/cluster-admin
/registry/clusterroles/edit
/registry/clusterroles/system:*
/registry/clusterroles/view
/registry/configmaps/kube-public/cluster-info
/registry/configmaps/kube-system/*
/registry/controllerrevisions/default/*
/registry/daemonsets/kube-system/kube-proxy
/registry/deployments/default/*
/registry/deployments/kube-system/*
/registry/events/kube-system/*
/registry/leases/kube-node-lease/minikube
/registry/masterleases/192.168.99.101
/registry/minions/minikube
/registry/namespaces/*
/registry/persistentvolumeclaims/default/*
/registry/persistentvolumes/*
/registry/pods/default/*
/registry/pods/kube-system/*
/registry/priorityclasses/*
/registry/ranges/serviceips
/registry/ranges/servicenodeports
/registry/replicasets/default/*
/registry/replicasets/kube-system/*
/registry/rolebindings/kube-public/*
/registry/rolebindings/kube-system/*
/registry/roles/kube-public/*
/registry/roles/kube-system/*
/registry/secrets/default/*
/registry/secrets/kube-node-lease/*
/registry/secrets/kube-public/*
/registry/secrets/kube-system/*
/registry/serviceaccounts/default/default
/registry/serviceaccounts/kube-node-lease/default
/registry/serviceaccounts/kube-public/default
/registry/serviceaccounts/kube-system/*
/registry/services/endpoints/default/*
/registry/services/endpoints/default/*
/registry/services/endpoints/kube-system/*
/registry/services/specs/default/*
/registry/statefulsets/default/*
/registry/storageclasses/*

To access the data directly in there is a blog https://jakubbujny.com/2018/09/02/what-stores-kubernetes-in-etcd/ which covers connecting to a Minikube Kubernetes etcd data store. The summary of the commands below copies the etcd certs from one pod to another, attaches to the pod which runs etcd and then uses the certs to make a connection and emit the keys:

➜  kubectl cp --namespace kube-system \
kube-apiserver-minikube:var/lib/minikube/certs/apiserver-etcd-client.crt \
apiserver-etcd-client.crt
➜  kubectl cp --namespace kube-system \
kube-apiserver-minikube:var/lib/minikube/certs/apiserver-etcd-client.key \
apiserver-etcd-client.key
➜  kubectl cp --namespace kube-system \
apiserver-etcd-client.crt \
etcd-minikube:var/lib/minikube/certs/
➜  kubectl cp --namespace kube-system \
apiserver-etcd-client.key \
etcd-minikube:var/lib/minikube/certs/
➜  kubectl exec -it --namespace kube-system etcd-minikube -- sh
/ # export ETCDCTL_API=3
/ # cd var/lib/minikube/certs/
/var/lib/minikube/certs # etcdctl \
--cacert="etcd/ca.crt" \
--key=apiserver-etcd-client.key \
--cert=apiserver-etcd-client.crt \
get / --prefix --keys-only

In larger Kubernetes deployments it is common to scale the number of Master Nodes. With the presence of etcd on the Master Nodes you should always scale to an odd number of nodes (1, 3, 5) so that etcd can quickly send back a commit. Commits require a majority of the nodes to acknowledge the command.

In the previous section, it was mentioned etcd could be fixed by deleting the etcd data store in Cloud Foundry, don’t do that on Kubernetes Master Nodes as there is no process to repopulate the data!

Backup the Source of Truth – Cloud Foundry

All kidding aside, as long as you backup the Cloud Foundry platform databases and have a smoking hole of a deployment, recovery involves redeploying Cloud Foundry with the last known good deployment manifest and then restoring the last available good database backups.

There is an important order of operations that needs to be noted: you can’t do a restore if you don’t have good backups to begin with. There are a few tools which exist to backup the Cloud Foundry databases:

• BOSH Backup Restore (BBR) – bosh-deployment and cf-deployment come with ops files which can be enabled to perform backups which are defined with those projects. The downside is you have to manage the retention of those backups manually. It also requires a partial outage to the CF API when the ccdb is backed up which is often a non-starter for production systems where this type of outage isn’t acceptable.
• SHIELD – A tool I’ve used with several customers which is traditionally deployed via BOSH. Starting with version 8 it has a simple interface for scheduling backups, retention, encryption and selecting where the backups are stored. You can schedule CF database backups without incurring any outages. A nifty trick is SHIELD can be scheduled to do BBR jobs.
• Ad hoc – I’ve already blogged about this topic. There are downsides to this approach since the operator is left with just a single onetime backup which must be manually managed, but it has its uses.

Backup the Source of Truth – Kubernetes

A few of the same tools for Cloud Foundry can be reused to backup the etcd database on the Master Node(s):

• Ad hoc – You can use etcdctl to generate a snapshot of a hot etcd cluster. Full details of doing this are in https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/. Add a bit of cron-foo and you can backup the cluster pretty easy.
• Heptio Velero (formerly Ark) – A popular tool for backing up etcd and the Persistent Volumes to S3-compatible backup storage providers. It provides a bit more flexibility in performing restores to other clusters and allowing the restores to come up a bit cleaner than a straight etcdctl snapshot save command.
• BOSH Backup Restore (BBR) – If you’ve used BOSH to deploy CFCR (formerly known as Kubo) you can use BBR to schedule the backup of the etcd data store.
• SHIELD – There will soon be a plugin to backup etcd, as long as the SHIELD Agent is configured and running on the Master Node you can schedule the backups and perform restores as needed.

Kubernetes Backups – The Missing Bits

Backing up etcd is not the only persistent data that is important to have to recover from a “smoking hole” scenario. Cloud Foundry’s nature is that it doesn’t have stateful applications. Kubernetes does not have this distinction. You can run stateful applications and store information with persistent volumes. As of this writing, I don’t know of a good way to backup persistent volumes natively (besides Velero). The best I’ve seen are creative solutions around using glusterfs, Portworx or an IaaS solution (GCP Persistent Disk, AWS EBS, Azure Storage). I’ll update this section if I find a better solution.

Next Steps

I have more reading and experimenting to do, I’ll be digging into connecting to containers and copying files and hope to have more soon. If there are other topics along the lines of “cf does x, how do I get kubernetes to do the same thing?” let me know in the comments below.

Thanks for reading!