Post 2, Electric Boogaloo
This is the second part of a series of posts about monitoring your CloudFoundry installations. You can find the first post in the series here. This post will cover some of the basics to encourage non Ruby developers to help us interface with services they’ve written.
So… You want a BOSH Monitor plugin?
We briefly touched on BOSH monitor in the first post. But the tl;dr is that BOSH Monitor runs plugins that fire whenever a NATS message is read from the bus. This allows us to forward those messages to a 3rd service. At this time there are documented solutions for
- Event Logger – Logs events
- PagerDuty – Sends events to PagerDuty.com
- DataDog – Sends various events to DataDog.com
- AWS CloudWatch – Sends various events to Amazon’s CloudWatch
- Emailer – Sends configurable Emails on events reciept
There are also a few undocumented plugins such as the NATS forwarder we mentioned in the previous post.
A Plugin by any other name
While there are many services to which you could reasonably forward this information there are only a few options at the moment. This may be because BOSH Monitor plugins are actually resident within the BOSH Monitor itself. There is not currently a plugin architecture that allows for including your feature from a separate source in to a running BOSH. Being that this is just a ruby application you can surely monkey patch your way in to Cloudfoundry monitoring glory but lets all agree not to do that. Thanks.
Your road to Open Source nirvana
Adding a plugin requires working directly on BOSH and hopefullly providing the team with a tidy pull request. Before getting started you’ll want to review the BOSH developer docs. A point to keep in mind is that they favor pull requests with small, single commits with obvious purpose. Adding a bosh monitor plugin requires touching a couple files and adding a couple more. So you should be able to keep your commit history nice and neat. For example, this is what my current commit looks like. And this is probably close to what yours should look like.
modified: lib/bosh/monitor.rb
new file: lib/bosh/monitor/plugins/<plugin>.rb
new file: spec/unit/bosh/monitor/plugins/<plugin>_spec.rb
modified: ../release/jobs/health_monitor/spec
modified: ../release/jobs/health_monitor/templates/health_monitor.yml.erb
Shut up and build something already!
With the pleasantries out of the way, lets get to work. If you are familiar with Ruby, this process should feel pretty familiar to you. If not, don’t run away, I think we can get through this.
First lets get some code and make sure it works. By checking out the bosh source and running it’s test suite.
git clone https://github.com/cloudfoundry/bosh.git
cd bosh
gem install bundler
bundle install
cd bosh-monitor
bundle exec rspec
If the test suite does not pass, you are welcome to do some spelunking in the code but if you think your environment is set up correctly it really should be passing. See the BOSH Developers user group for help.
Welcome, Ruby Expert!
The lines above will get you far enough to poke around and get familiar with BOSH Monitor. If you don’t know Git, Ruby or Bundler this is your chance to consult Google. Go ahead.. We’ll wait.
To get started we’ll need to create two files in BOSH-Monitor.
lib/bosh/monitor/plugins/<plugin>.rb and spec/unit/bosh/monitor/plugins/<plugin>_spec.rb This is the bulk of your work.
Anatomy of a Plugin
The first thing to keep in mind is that your plugin will run in Event Machine and the basic template below will get you started.
module Bosh::Monitor
module Plugins
class MyPlugin < Base
#You should override validate options in your plugin
#It should return true or false depending on the validity of options
#provided in the manifest
def validate_options
true
end
#Your run method controls the overall state of the event loop and
#it will get called after the plugin instance is initialized.
#Here we can parse out options or set up global plugin state
def run
#Example, initialize option in to instance
@option_one = options['option_one']
#Example, Announce your plugin's existence
logger.info("My plugin is running...")
end
#Your process method will be called whenever a new event is found
#on the Nats message bus.
def process(event)
#Example. You can use the event type to specify how you'd like to parse
#a given message
case event
when Bosh::Monitor::Events::Heartbeat
#Forward to your service
when Bosh::Monitor::Events::Alert
#Forward to your service
else
#Do something or Do nothing, It's Anarchy!
end
end
end
end
end
Note that if you are writing a plugin that needs to put or post to a 3rd party service there is a library included in bosh-monitor/lib/bosh/monitor/plugins/http_request_helper.rb You probably want to use or extend that module instead rolling your own.
Testing 1,2,3..
Testing may the most complicated part of writing your plugin. Fortunately there are lots of examples in the bosh-monitor/spec/unit/bosh/monitor/plugins. The example below provides a couple of tests to ensure your plugin hands an event off to your service and that it validates options appropriately. It assumes you are using the http_request_helper module. These are the very basic tests you will likely need.
require 'spec_helper'
describe Bhm::Plugins::MyPlugin do
#set subject to a new instantiated bosh monitor plugin of your type
subject{ described_class.new(options) }
#When you instantiate a plugin you will pass it required options
let(:options){ { 'option1' => 'foo', 'option2' => 'bar' }}
#make_heartbeat is a helper that creats a fake event for you to process
let(:heartbeat){ make_heartbeat(timestamp: 1320196099) }
#A fake uri for your service endpoint
let(:uri){ URI.parse("http://fake-service-endpoint/path/to/service") }
#What is the body of the request to your service going to look like
#In this case we are just fowarding the heartbeat directly
let(:request){ { :body => heartbeat.to_json } }
context "with invalid options" do
let(:options){{ }}
it "is not valid" do
subject.run
expect(subject.validate_options).to eq(false)
end
end
context "with valid options" do
it "is valid" do
subject.run
expect(subject.validate_options).to eq(true)
end
#Ensure that your plugin hands the package off to your service
it "forwards events to service" do
#calling run on your plugin instance will begin its event loop
subject.run
expect(subject).to_not receive(:send_http_put_request).with(uri, request)
subject.process(heartbeat)
end
end
end
end
Trading Options
The two files we havent touched yet are configuration and template files that tell BOSH how to pass our options in to the plugin.
First we have /release/jobs/health_monitor/templates/health_monitor.yml.erb which is a template that tells BOSH how to apply options from spiff templates. Note that option1 is required as it is not wrapped in an if_p block. Deploying BOSH with consul_enabled, without this set will result in an error. But option2 will be skipped if it does not exist in your manifest.
...
<% if p("hm.my_plugin_enabled") %>
- name: my_plugin
events:
- alert
- heartbeat
options:
option1: <%= p("hm.my_plugin.option1") %>
<% if_p("hm.my_plugin.option2") %>
option2: <%= p("hm.my_plugin.option2") %>
<%end%>
...
<% end %>
...
And lastly we have /release/jobs/health_monitor/spec which defines all of your options for BOSH.
hm.my_plugin_enabled:
description: Enable Your Service
default: false
hm.my_plugin.option1:
description: What is option 1?
hm.my_plugin.option2:
description: What is option2?
That should be enough to get you started. You can run your tests from the bosh-monitor directory with bundle exec rspec and you can get more help here
Finally, once you have a nice simple plugin with passing tests you’ll want to test your plugin in a real world example.
Run the rake task to create a bosh release of your modified bosh
bundle exec rake release:create_dev_release
This will muddy up your local branch with release details so you may want to commit your current branch and create a new playground branch for the deployment.
Now upload to your microbosh
bundle exec rake release:upload_dev_release
Now you can get your dev release version by running bosh releases
Create a manifest file and add your plugin options to the health monitor section
properties:
hm:
my_plugin_enabled: true
my_plugin:
option1: 'foo'
option2: 'bar'
Now deploy your BOSH and test your plugin in a real setting.
bosh deployment path/to/manifest.yml
bosh deploy
You’ll want to deploy something non trivial to the BOSH in order to generate some nats traffic for monitoring. Since this is a series about monitoring Cloud Foundry. We’d recommend starting there.
Next up we’ll be releasing announcing a new plugin. Stay tuned.
Spread the word
