Apr 02, 2018 Deploy HA CF with Anti-Affinity DRS Rules in vSphere
VM-VM Affinity Rules
vSphere VM-VM Affinity Rules specify whether selected individual VMs should run on the same host or be kept on separate hosts. You can either set Distribute Resource Scheduler (DRS) Affinity Rule which will keep all the VMs on the same host or DRS Anti-Affinity Rule which requires that each VM is on its own host. DRS Anti-Affinity Rule can be used to achieve host level High Availability (HA); since when you put VMs that run the same jobs on different hosts, if one host is down, you still have other nodes running and working.
Note, the number of the hosts must be equal to or greater than the number of vms you want to place with anti-affinity Distributed Resource Scheduler (DRS) rule.
HA in CF Deployment
To achieve HA in CF deployment, you can horizontally scale most Cloud Foundry components to multiple instances into different Availability Zones (Azs), in this scenario, onto different ESXi hosts. Using DRS Anti-Affinity Rule to achieve host level HA has two limitations: one is that the host number must not be less than the number of instances you want to apply DRS Anti-Affinity Rule; the other is you have to set DRS Anti-Affinity Rules for each type of instances you want to achieve host level HA while you should not create too many Anti-Affinity Rules in one cluster.
You could also use vSphere Resource Poolto achieve host level HA. You have to deal with load balance among all the hosts since the number of instances for different CF component can be 2, 3, and many more. How to use vSphere Resource Pool to achieve HA in CF is out of this blog's scope.
Let's continue to talk about how to use DRS Anti-Affinity Rule in CF deployment. We will use the
ectd_consul instances in CF deployment as an example to show you how to configure VM-VM anti-affinity DRS rule to make sure each
ectd_consul node is on its own host.
Before we can apply Anti-Affinity DRS Rule to the
consul_etcd job in CF deployment, we need enable DRS Automation for the cluster that we are deploying vms in.
Enable DRS Automation
Go to your vSphere Web Client, right click on the cluster, choose
setting, then click
vSphere DRS, turn on the DRS automation and pick automation level such as
Fully Automation or
Next, we need to configure
vm_type with Anti-Affinity DRS Rule in cloud config and apply it to the
consul_etcd job in CF deployment manifest.
Define vm_type with Anti-Affinity DRS Rule in Cloud Config
In your cloud config, define a
consul_etcd as follows. This adds Anti-Affinity DRS rule to the vSphere cluster with a
separate_vms rule type.
vm_type: - name: consul_etcd cloud_properties: datacenters: - name: my-dc clusters: - my-vsphere-cluster: drs_rules: - name: separate-consul-etcd-rule type: separate_vms
In the manifest to deploy CF, configure the
consul_etcd job in the instance groups to use the
vm_type. In a deployment repo generated using
genesis, you can add the following configuration in you environment yml file. This configuration will create 3 VMs onto 3 different ESXi hosts in the cluster specified in the above cloud config. Without DRS Anti-Affinity Rule, quite often it creates 3 VMs onto 2 hosts even the total number of hosts is 3.
instance_groups: - name: consul_etcd instances: 3 vm_type: consul_etcd persistent_disk_type: consul networks: - your_consul_network_in_cc jobs: - name: etcd release: ectd properties: etcd: ... - name: metron_agent ....
Now you are ready to deploy. You will see the 3 consul nodes are allocated on 3 different ESXi hosts.