Aug 01, 2014 Making Support happy: aggregating Cloud Foundry component logs
It was three full days into a five day training course when suddenly the students - Support staff - became very animated and excited.
They were watching all the aggregated Cloud Foundry component logs & events from across the entire system.
"Oh I wish you'd shown us this on day one!"
Ahh blessed Support people can be more excited about what's wrong with a system than the incredible discovery that something like Cloud Foundry even works at all.
Cloud Foundry includes two layers of logging:
- user/application logs and events
- operations component logs and message bus events
The former are expressly for the end users/teams. These logs can be streamed to the local command line or continously drained off to logging services like Papertrail or Logentries.
The latter was what the Support training students had seen. All the components' logs (router, cloudcontroller, health manager, runner, etc) and all the intercommunication messages (message bus) were being aggregated and stored.
In the demo to the students, and the video below, we use the built-in example aggregator that comes bundled with Cloud Foundry. In production, most operations people connect in Logstash or Splunk or similar.
Enabling syslog for components
Whether you use the built-in example aggregator job or a remote syslog receiver such as Logstash or Splunk, you want to add the following global property to your BOSH deployment manifest for Cloud Foundry.
properties: syslog_aggregator: address: 10.15.213.19 port: 54321
port values are for the inbound endpoint of your logging aggregator.
tcp transport is used. You can change it to
properties: syslog_aggregator: address: 10.15.213.19 port: 54321 transport: udp
Running the example aggregator
In the video above we run the
syslog_aggregator job that is built-in with Cloud Foundry's BOSH release.
Example configuration for running it could be:
jobs: - name: syslog_aggregator_z1 templates: - name: syslog_aggregator release: cf instances: 1 resource_pool: small_z1 networks: - name: cf1 default: [dns, gateway] static_ips: - 10.15.213.19
static_ip should be used by the
properties.syslog_aggregator.address configuration in the previous section.