Demo of a manual failover of redis cluster atop of consul/confd

The animated gif* demo below shows:

1. observe a master node failure (consul health checks)
2. promote a slave to become master
3. configure other slaves to the new master
4. configure the original master to become a slave (if it ever awakens/returns to the cluster)

And all without touching my configuration management tool (which is BOSH in this example).

When you configure your services (such as redis) using Chef/Puppet/BOSH, those services aren’t really capable of being dynamically healed – that is, a slave being promoted to master, and the master being killed or forced to being a slave (if it should return to the cluster).

While the demo shows the failover steps as manual steps, ultimately my dream is for the failover to be automatic.

The demo shows that the programmatic levers are available for failover. Change the configuration in consul’s key-value store and you change the live cluster configuration.

This demo is using:

• consul for service discovery (bosh consul services was displaying the available services) and for the health dashboard
• confd to dynamically create configuration files based on consul key-value (kv) values
• BOSH to provision the 3-node cluster, package management and bootstrapping configuration management

BOSH projects

• redis-boshrelease v8 has been released and is demoed above

• consul-boshrelease v4 has been released and includes the lovely consul UI showed in the demo

• OMG I love using LICEcap