Here is a Pager Duty alert for a persistent disk being greater than 80% full on an etcd node:
It turned out to be relatively simple to "turn on" disk monitoring on all jobs that contained persistent disks, across all deployments, across all stages of our client’s pipeline.
We created a simple BOSH release that just adds some monit
monitoring; and BOSH health monitor does the rest. It was all built-in.
Introducing monitor-server
https://github.com/cloudfoundry-community/monitor-server-boshrelease
This BOSH release initially includes monitoring of ephemeral and persistent disks on a BOSH VM. Conceptually it could monitor for any global attribute of a VM that good ol’ monit can monitor
[See Quick guide to using Monit in BOSH for an introduction to Monit in BOSH]
By default it alerts on either disks reaching 80% capacity.
See the monitor-server README for how to co-locate it with all your BOSH deployments for instant disk monitoring.
BOSH health alerts
BOSH already includes a Health Monitor that forwards any alerts from any VM’s monit
monitor. All you have to do is turn it on.
Currently the following backends are supported:
- SMTP/email
- Pager Duty
- DataDog
- AWS CloudWatch
Read the BOSH docs for more information on monitoring: