Stark & Wayne

Mental Migration: A Cloud Foundry Operator's Perspective on Starting Kubernetes - Part 2

This is the second part of my journey as a Cloud Foundry operator learning about the various components Kubernetes by comparing the components to functionalities I know in Cloud Foundry.  The start of this series is here: https://starkandwayne.com/blog/mental-migration-a-cf-operators-perspective-on-starting-kubernetes/

Source of Truth - Cloud Foundry

In Cloud Foundry land there are relational databases that contain the desired state of the platform.  If you have your PostgreSQL/MySQL databases for the Cloud Controller, UAA, Diego, Silk, and Locket as externally deployed resources, you can completely delete all the Cloud Foundry vms and deployment, count to 100, redeploy CF via BOSH. Other than the Change Control Board representative standing at your desk giving you dirty looks (Hi Jean!), CF will come back up happily and continue to serve up apps.

If you start to dig into the Databases of Cloud Foundry, especially the Cloud Controller Database (ccdb), the desired state of the platform is represented across a few tables.  There are instructions here for connecting directly to the database. Information on the desired state of apps, orgs, spaces, processes, quotas, and services are stored here.  There are a few interesting queries documented at here and here which can help you gain some insights on your CF deployment.

Below is a list of the tables in the ccdb to give you a rough idea of the objects which are tracked:

app_events
security_groups
app_usage_events
apps
apps_routes
apps_v3
billing_events
buildpack_lifecycle_data
buildpacks
delayed_jobs
domains
droplets
env_groups
events
feature_flags
isolation_segments
lockings
organizations
organizations_auditors
organizations_billing_managers
organizations_managers
organizations_private_domains
organizations_users
package_docker_data
packages
quota_definitions
route_bindings
route_mappings
routes
schema_migrations
security_groups_spaces
service_auth_tokens
service_bindings
service_brokers
service_dashboard_clients
service_instance_operations
service_instances
service_keys
service_plan_visibilities
service_plans
service_usage_events
services
snw_delayed_jobs_backup
space_quota_definitions
spaces
spaces_auditors
spaces_developers
spaces_managers
stacks
tasks
users
v3_droplets
v3_service_bindings

If you have been using Cloud Foundry long enough you'll remember a time where etcd was used to store "current state". Upgrades at times became "challenging" because of the number of times etcd would simply refuse to come up clean, this blog post was used several times to glue large deployments back together, typically after 3 am.  In short, you would wipe the etcd data store and let CF repopulate the data.

Source of Truth - Kubernetes

Instead of a relational database like PostgreSQL or MySQL, Kubernetes uses etcd to store information about the clusters.  Desired and current state is stored in etcd. Information such as secrets, resources, pods, deployments, services and networking definitions are stored in the etcd data store.

Like Cloud Foundry, Kubernetes uses an API layer to abstract direct access to the etcd data store. Accessing the data indirectly is done with the cf and kubectl CLIs.

Below is a summary of the keys on a minikube Kubernetes deployment:

/registry/apiregistration.k8s.io/apiservices/v1/*
/registry/clusterrolebindings/cluster-admin
/registry/clusterrolebindings/kubeadm:*
/registry/clusterrolebindings/minikube-rbac
/registry/clusterrolebindings/storage-provisioner
/registry/clusterrolebindings/system:*
/registry/clusterroles/admin
/registry/clusterroles/cluster-admin
/registry/clusterroles/edit
/registry/clusterroles/system:*
/registry/clusterroles/view
/registry/configmaps/kube-public/cluster-info
/registry/configmaps/kube-system/*
/registry/controllerrevisions/default/*
/registry/daemonsets/kube-system/kube-proxy
/registry/deployments/default/*
/registry/deployments/kube-system/*
/registry/events/kube-system/*
/registry/leases/kube-node-lease/minikube
/registry/masterleases/192.168.99.101
/registry/minions/minikube
/registry/namespaces/*
/registry/persistentvolumeclaims/default/*
/registry/persistentvolumes/*
/registry/pods/default/*
/registry/pods/kube-system/*
/registry/priorityclasses/*
/registry/ranges/serviceips
/registry/ranges/servicenodeports
/registry/replicasets/default/*
/registry/replicasets/kube-system/*
/registry/rolebindings/kube-public/*
/registry/rolebindings/kube-system/*
/registry/roles/kube-public/*
/registry/roles/kube-system/*
/registry/secrets/default/*
/registry/secrets/kube-node-lease/*
/registry/secrets/kube-public/*
/registry/secrets/kube-system/*
/registry/serviceaccounts/default/default
/registry/serviceaccounts/kube-node-lease/default
/registry/serviceaccounts/kube-public/default
/registry/serviceaccounts/kube-system/*
/registry/services/endpoints/default/*
/registry/services/endpoints/default/*
/registry/services/endpoints/kube-system/*
/registry/services/specs/default/*
/registry/statefulsets/default/*
/registry/storageclasses/*

To access the data directly in there is a blog https://jakubbujny.com/2018/09/02/what-stores-kubernetes-in-etcd/ which covers connecting to a Minikube Kubernetes etcd data store.  The summary of the commands below copies the etcd certs from one pod to another, attaches to the pod which runs etcd and then uses the certs to make a connection and emit the keys:

➜  kubectl cp --namespace kube-system \
kube-apiserver-minikube:var/lib/minikube/certs/apiserver-etcd-client.crt \
apiserver-etcd-client.crt
➜  kubectl cp --namespace kube-system \
kube-apiserver-minikube:var/lib/minikube/certs/apiserver-etcd-client.key \
apiserver-etcd-client.key
➜  kubectl cp --namespace kube-system \
apiserver-etcd-client.crt \
etcd-minikube:var/lib/minikube/certs/
➜  kubectl cp --namespace kube-system \
apiserver-etcd-client.key \
etcd-minikube:var/lib/minikube/certs/
➜  kubectl exec -it --namespace kube-system etcd-minikube -- sh
/ # export ETCDCTL_API=3
/ # cd var/lib/minikube/certs/

/var/lib/minikube/certs # etcdctl \
--cacert="etcd/ca.crt" \
--key=apiserver-etcd-client.key \
--cert=apiserver-etcd-client.crt \
get / --prefix --keys-only

In larger Kubernetes deployments it is common to scale the number of Master Nodes.  With the presence of etcd on the Master Nodes you should always scale to an odd number of nodes (1, 3, 5) so that etcd can quickly send back a commit.  Commits require a majority of the nodes to acknowledge the command.

In the previous section, it was mentioned etcd could be fixed by deleting the etcd data store in Cloud Foundry, don't do that on Kubernetes Master Nodes as there is no process to repopulate the data!

Backup the Source of Truth - Cloud Foundry

All kidding aside, as long as you backup the Cloud Foundry platform databases and have a smoking hole of a deployment, recovery involves redeploying Cloud Foundry with the last known good deployment manifest and then restoring the last available good database backups.

There is an important order of operations that needs to be noted: you can't do a restore if you don't have good backups to begin with.  There are a few tools which exist to backup the Cloud Foundry databases:

Backup the Source of Truth - Kubernetes

A few of the same tools for Cloud Foundry can be reused to backup the etcd database on the Master Node(s):

Kubernetes Backups - The Missing Bits

Backing up etcd is not the only persistent data that is important to have to recover from a "smoking hole" scenario.  Cloud Foundry's nature is that it doesn't have stateful applications.  Kubernetes does not have this distinction.  You can run stateful applications and store information with persistent volumes.  As of this writing, I don't know of a good way to backup persistent volumes natively (besides Velero).  The best I've seen are creative solutions around using glusterfs, Portworx or an IaaS solution (GCP Persistent Disk, AWS EBS, Azure Storage).  I'll update this section if I find a better solution.

Next Steps

I have more reading and experimenting to do, I'll be digging into connecting to containers and copying files and hope to have more soon.  If there are other topics along the lines of "cf does x, how do I get kubernetes to do the same thing?" let me know in the comments below.

Thanks for reading!