Deploy HA CF with Anti-Affinity DRS Rules in vSphere

VM-VM Affinity Rules

vSphere VM-VM Affinity Rules specify whether selected individual VMs should run on the same host or be kept on separate hosts. You can either set Distribute Resource Scheduler (DRS) Affinity Rule which will keep all the VMs on the same host or DRS Anti-Affinity Rule which requires that each VM is on its own host. DRS Anti-Affinity Rule can be used to achieve host level High Availability (HA); since when you put VMs that run the same jobs on different hosts, if one host is down, you still have other nodes running and working.

Note, the number of the hosts must be equal to or greater than the number of vms you want to place with anti-affinity Distributed Resource Scheduler (DRS) rule.

HA in CF Deployment

To achieve HA in CF deployment, you can horizontally scale most Cloud Foundry components to multiple instances into different Availability Zones (Azs), in this scenario, onto different ESXi hosts. Using DRS Anti-Affinity Rule to achieve host level HA has two limitations: one is that the host number must not be less than the number of instances you want to apply DRS Anti-Affinity Rule; the other is you have to set DRS Anti-Affinity Rules for each type of instances you want to achieve host level HA while you should not create too many Anti-Affinity Rules in one cluster.

You could also use vSphere Resource Poolto achieve host level HA. You have to deal with load balance among all the hosts since the number of instances for different CF component can be 2, 3, and many more. How to use vSphere Resource Pool to achieve HA in CF is out of this blog’s scope.

Let’s continue to talk about how to use DRS Anti-Affinity Rule in CF deployment. We will use the ectd_consul instances in CF deployment as an example to show you how to configure VM-VM anti-affinity DRS rule to make sure each ectd_consul node is on its own host.

Before we can apply Anti-Affinity DRS Rule to the consul_etcd job in CF deployment, we need enable DRS Automation for the cluster that we are deploying vms in.

Enable DRS Automation

Go to your vSphere Web Client, right click on the cluster, choose setting, then click vSphere DRS, turn on the DRS automation and pick automation level such as Fully Automation or Partially Automation.

Next, we need to configure vm_type with Anti-Affinity DRS Rule in cloud config and apply it to the consul_etcd job in CF deployment manifest.

Define vm_type with Anti-Affinity DRS Rule in Cloud Config

In your cloud config, define a vm_type named consul_etcd as follows. This adds Anti-Affinity DRS rule to the vSphere cluster with a separate_vms rule type.

vm_type:
- name: consul_etcd
  cloud_properties:
    datacenters:
    - name: my-dc
      clusters:
      - my-vsphere-cluster:
          drs_rules:
          - name: separate-consul-etcd-rule
            type: separate_vms

Configure Manifest

In the manifest to deploy CF, configure the consul_etcd job in the instance groups to use the consul_etcd as vm_type. In a deployment repo generated using genesis, you can add the following configuration in you environment yml file. This configuration will create 3 VMs onto 3 different ESXi hosts in the cluster specified in the above cloud config. Without DRS Anti-Affinity Rule, quite often it creates 3 VMs onto 2 hosts even the total number of hosts is 3.

instance_groups:
- name: consul_etcd
  instances: 3
  vm_type: consul_etcd
  persistent_disk_type: consul
  networks:
  - your_consul_network_in_cc
  jobs:
    - name: etcd
      release: ectd
      properties:
        etcd:
        ...
    - name: metron_agent
    ....

Now you are ready to deploy. You will see the 3 consul nodes are allocated on 3 different ESXi hosts.