Stark & Wayne

Demo of a manual failover of redis cluster atop of consul/confd

The animated gif* demo below shows:

  1. observe a master node failure (consul health checks)
  2. promote a slave to become master
  3. configure other slaves to the new master
  4. configure the original master to become a slave (if it ever awakens/returns to the cluster)

And all without touching my configuration management tool (which is BOSH in this example).

cluster failover

When you configure your services (such as redis) using Chef/Puppet/BOSH, those services aren't really capable of being dynamically healed - that is, a slave being promoted to master, and the master being killed or forced to being a slave (if it should return to the cluster).

While the demo shows the failover steps as manual steps, ultimately my dream is for the failover to be automatic.

The demo shows that the programmatic levers are available for failover. Change the configuration in consul's key-value store and you change the live cluster configuration.

This demo is using:

BOSH projects