Making Support happy: aggregating Cloud Foundry component logs

It was three full days into a five day training course when suddenly the students – Support staff – became very animated and excited.

They were watching all the aggregated Cloud Foundry component logs & events from across the entire system.

"Oh I wish you’d shown us this on day one!"

Ahh blessed Support people can be more excited about what’s wrong with a system than the incredible discovery that something like Cloud Foundry even works at all.

Cloud Foundry includes two layers of logging:

• user/application logs and events
• operations component logs and message bus events

The former are expressly for the end users/teams. These logs can be streamed to the local command line or continously drained off to logging services like Papertrail or Logentries.

The latter was what the Support training students had seen. All the components’ logs (router, cloudcontroller, health manager, runner, etc) and all the intercommunication messages (message bus) were being aggregated and stored.

In the demo to the students, and the video below, we use the built-in example aggregator that comes bundled with Cloud Foundry. In production, most operations people connect in Logstash or Splunk or similar.

Enabling syslog for components

Whether you use the built-in example aggregator job or a remote syslog receiver such as Logstash or Splunk, you want to add the following global property to your BOSH deployment manifest for Cloud Foundry.

properties:
  syslog_aggregator:
    address: 10.15.213.19
    port: 54321

Where the address and port values are for the inbound endpoint of your logging aggregator.

By default tcp transport is used. You can change it to udp or repl:

properties:
  syslog_aggregator:
    address: 10.15.213.19
    port: 54321
    transport: udp

Running the example aggregator

In the video above we run the syslog_aggregator job that is built-in with Cloud Foundry’s BOSH release.

Example configuration for running it could be:

jobs:
  - name: syslog_aggregator_z1
    templates:
      - name: syslog_aggregator
        release: cf
    instances: 1
    resource_pool: small_z1
    networks:
      - name: cf1
        default: [dns, gateway]
        static_ips:
          - 10.15.213.19

The assigned static_ip should be used by the properties.syslog_aggregator.address configuration in the previous section.