May 24, 2019 Mental Migration: A Cloud Foundry Operator's Perspective on Starting Kubernetes - Part 2
This is the second part of my journey as a Cloud Foundry operator learning about the various components Kubernetes by comparing the components to functionalities I know in Cloud Foundry. The start of this series is here: https://starkandwayne.com/blog/mental-migration-a-cf-operators-perspective-on-starting-kubernetes/
Source of Truth - Cloud Foundry
In Cloud Foundry land there are relational databases that contain the desired state of the platform. If you have your PostgreSQL/MySQL databases for the Cloud Controller, UAA, Diego, Silk, and Locket as externally deployed resources, you can completely delete all the Cloud Foundry vms and deployment, count to 100, redeploy CF via BOSH. Other than the Change Control Board representative standing at your desk giving you dirty looks (Hi Jean!), CF will come back up happily and continue to serve up apps.
If you start to dig into the Databases of Cloud Foundry, especially the Cloud Controller Database (
ccdb), the desired state of the platform is represented across a few tables. There are instructions here for connecting directly to the database. Information on the desired state of apps, orgs, spaces, processes, quotas, and services are stored here. There are a few interesting queries documented at here and here which can help you gain some insights on your CF deployment.
Below is a list of the tables in the
ccdb to give you a rough idea of the objects which are tracked:
app_events security_groups app_usage_events apps apps_routes apps_v3 billing_events buildpack_lifecycle_data buildpacks delayed_jobs domains droplets env_groups events feature_flags isolation_segments lockings organizations organizations_auditors organizations_billing_managers organizations_managers organizations_private_domains organizations_users package_docker_data packages quota_definitions route_bindings route_mappings routes schema_migrations security_groups_spaces service_auth_tokens service_bindings service_brokers service_dashboard_clients service_instance_operations service_instances service_keys service_plan_visibilities service_plans service_usage_events services snw_delayed_jobs_backup space_quota_definitions spaces spaces_auditors spaces_developers spaces_managers stacks tasks users v3_droplets v3_service_bindings
If you have been using Cloud Foundry long enough you'll remember a time where
etcd was used to store "current state". Upgrades at times became "challenging" because of the number of times
etcd would simply refuse to come up clean, this blog post was used several times to glue large deployments back together, typically after 3 am. In short, you would wipe the
etcd data store and let CF repopulate the data.
Source of Truth - Kubernetes
Instead of a relational database like PostgreSQL or MySQL, Kubernetes uses
etcd to store information about the clusters. Desired and current state is stored in
etcd. Information such as secrets, resources, pods, deployments, services and networking definitions are stored in the
etcd data store.
Like Cloud Foundry, Kubernetes uses an API layer to abstract direct access to the
etcd data store. Accessing the data indirectly is done with the
Below is a summary of the keys on a
minikube Kubernetes deployment:
/registry/apiregistration.k8s.io/apiservices/v1/* /registry/clusterrolebindings/cluster-admin /registry/clusterrolebindings/kubeadm:* /registry/clusterrolebindings/minikube-rbac /registry/clusterrolebindings/storage-provisioner /registry/clusterrolebindings/system:* /registry/clusterroles/admin /registry/clusterroles/cluster-admin /registry/clusterroles/edit /registry/clusterroles/system:* /registry/clusterroles/view /registry/configmaps/kube-public/cluster-info /registry/configmaps/kube-system/* /registry/controllerrevisions/default/* /registry/daemonsets/kube-system/kube-proxy /registry/deployments/default/* /registry/deployments/kube-system/* /registry/events/kube-system/* /registry/leases/kube-node-lease/minikube /registry/masterleases/192.168.99.101 /registry/minions/minikube /registry/namespaces/* /registry/persistentvolumeclaims/default/* /registry/persistentvolumes/* /registry/pods/default/* /registry/pods/kube-system/* /registry/priorityclasses/* /registry/ranges/serviceips /registry/ranges/servicenodeports /registry/replicasets/default/* /registry/replicasets/kube-system/* /registry/rolebindings/kube-public/* /registry/rolebindings/kube-system/* /registry/roles/kube-public/* /registry/roles/kube-system/* /registry/secrets/default/* /registry/secrets/kube-node-lease/* /registry/secrets/kube-public/* /registry/secrets/kube-system/* /registry/serviceaccounts/default/default /registry/serviceaccounts/kube-node-lease/default /registry/serviceaccounts/kube-public/default /registry/serviceaccounts/kube-system/* /registry/services/endpoints/default/* /registry/services/endpoints/default/* /registry/services/endpoints/kube-system/* /registry/services/specs/default/* /registry/statefulsets/default/* /registry/storageclasses/*
To access the data directly in there is a blog https://jakubbujny.com/2018/09/02/what-stores-kubernetes-in-etcd/ which covers connecting to a Minikube Kubernetes
etcd data store. The summary of the commands below copies the etcd certs from one pod to another, attaches to the pod which runs
etcd and then uses the certs to make a connection and emit the keys:
➜ kubectl cp --namespace kube-system \ kube-apiserver-minikube:var/lib/minikube/certs/apiserver-etcd-client.crt \ apiserver-etcd-client.crt ➜ kubectl cp --namespace kube-system \ kube-apiserver-minikube:var/lib/minikube/certs/apiserver-etcd-client.key \ apiserver-etcd-client.key ➜ kubectl cp --namespace kube-system \ apiserver-etcd-client.crt \ etcd-minikube:var/lib/minikube/certs/ ➜ kubectl cp --namespace kube-system \ apiserver-etcd-client.key \ etcd-minikube:var/lib/minikube/certs/ ➜ kubectl exec -it --namespace kube-system etcd-minikube -- sh / # export ETCDCTL_API=3 / # cd var/lib/minikube/certs/ /var/lib/minikube/certs # etcdctl \ --cacert="etcd/ca.crt" \ --key=apiserver-etcd-client.key \ --cert=apiserver-etcd-client.crt \ get / --prefix --keys-only
In larger Kubernetes deployments it is common to scale the number of Master Nodes. With the presence of
etcd on the Master Nodes you should always scale to an odd number of nodes (1, 3, 5) so that
etcd can quickly send back a commit. Commits require a majority of the nodes to acknowledge the command.
In the previous section, it was mentioned etcd could be fixed by deleting the
etcd data store in Cloud Foundry, don't do that on Kubernetes Master Nodes as there is no process to repopulate the data!
Backup the Source of Truth - Cloud Foundry
All kidding aside, as long as you backup the Cloud Foundry platform databases and have a smoking hole of a deployment, recovery involves redeploying Cloud Foundry with the last known good deployment manifest and then restoring the last available good database backups.
There is an important order of operations that needs to be noted: you can't do a restore if you don't have good backups to begin with. There are a few tools which exist to backup the Cloud Foundry databases:
- BOSH Backup Restore (BBR) -
cf-deploymentcome with ops files which can be enabled to perform backups which are defined with those projects. The downside is you have to manage the retention of those backups manually. It also requires a partial outage to the CF API when the
ccdbis backed up which is often a non-starter for production systems where this type of outage isn't acceptable.
- SHIELD - A tool I've used with several customers which is traditionally deployed via BOSH. Starting with version 8 it has a simple interface for scheduling backups, retention, encryption and selecting where the backups are stored. You can schedule CF database backups without incurring any outages. A nifty trick is SHIELD can be scheduled to do BBR jobs.
- Ad hoc - I've already blogged about this topic. There are downsides to this approach since the operator is left with just a single onetime backup which must be manually managed, but it has its uses.
Backup the Source of Truth - Kubernetes
A few of the same tools for Cloud Foundry can be reused to backup the
etcd database on the Master Node(s):
- Ad hoc - You can use
etcdctlto generate a snapshot of a hot
etcdcluster. Full details of doing this are in https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/. Add a bit of cron-foo and you can backup the cluster pretty easy.
- Heptio Velero (formerly Ark) - A popular tool for backing up
etcdand the Persistent Volumes to S3-compatible backup storage providers. It provides a bit more flexibility in performing restores to other clusters and allowing the restores to come up a bit cleaner than a straight
etcdctl snapshot savecommand.
- BOSH Backup Restore (BBR) - If you've used BOSH to deploy CFCR (formerly known as Kubo) you can use BBR to schedule the backup of the
- SHIELD - There will soon be a plugin to backup
etcd, as long as the SHIELD Agent is configured and running on the Master Node you can schedule the backups and perform restores as needed.
Kubernetes Backups - The Missing Bits
etcd is not the only persistent data that is important to have to recover from a "smoking hole" scenario. Cloud Foundry's nature is that it doesn't have stateful applications. Kubernetes does not have this distinction. You can run stateful applications and store information with persistent volumes. As of this writing, I don't know of a good way to backup persistent volumes natively (besides Velero). The best I've seen are creative solutions around using glusterfs, Portworx or an IaaS solution (GCP Persistent Disk, AWS EBS, Azure Storage). I'll update this section if I find a better solution.
I have more reading and experimenting to do, I'll be digging into connecting to containers and copying files and hope to have more soon. If there are other topics along the lines of "cf does x, how do I get kubernetes to do the same thing?" let me know in the comments below.
Thanks for reading!