Multi-tenant ELK for your private Cloud Foundry

If you’re using a public Cloud Foundry – Pivotal Web Services, IBM Blue Mix – then you are probably ok with the idea of using a public log storage service like Papertrail or Logentries. What about if you are using your own Cloud Foundry – open source or PivotalCF – and you’re not allowed to use a hosted log service? What are your options?

Our various clients have needed something and I thought I’d share one idea. It has pros and cons.

Pretty?

Events over time:

Line-by-line logs

Pros and cons

The pros are:

• using well known Elastic Search/Logstash/Kibana stack [ELK]
• it exists as a solution
• it allows multi-tenancy isolation of logs
• easy deployment via bosh-workspace for deploying docker/logstash as a service
• all components are open source

The cons are:

• it uses the relatively unmaintained cf-containers-broker project
• not easily horizontally scalable – the cf-containers-broker project only supports a single Docker server
• each logstash service instance is running in a single Docker container and cannot be scaled out
• limited ops tooling to manage the logstash docker containers once they are instantiated
• the kibana app includes an insecure proxy through to elastic search

But, above all the cons are the pros – it exists and it is multi-tenant.

Perhaps the solution will work for you long enough until a better solution is proposed (hopefully by us, but we’d love any solution).

Why not logsearch?

http://www.logsearch.io/ is a distribution of ELK that include a BOSH release and BOSH workspace and serves as a great backend for Cloud Foundry component logs.

And with a firehose nozzle [see firehose-to-syslog] for loggregator it could also store all applications’ logs.

But there is no multi-tenancy for Kibana – the UI portion of ELK – to restrict what a user can see. We only want a user to see the logs that a) they want to see; b) they have permission to see.

Elastic, the company, does have a product Shield that might solve this problem. I learnt about this yesterday and will investigate it later.

How do we do multi-tenant ELK?

Two components:

• cf-containers-broker is a Cloud Foundry service broker that locally provisions Docker containers – we can use it to run ELK in a Docker container
• kibana-me-logs is an app that shows the Kibana UI to display your application’s logs

A platform engineer will deploy the former component once and it will be used by all users as a Cloud Foundry service.

Each application developer will deploy the latter app so they can view their logstash service instance.

This solution means that you application’s logs are isolated inside a dedicated Docker container that runs logstash and elastic search.

To view the logs via Kibana you run the kibana-me-logs app as another Cloud Foundry app.

The images above are examples.

Coming soon

Currently I’m working on a way to make it very simple for platform engineers to deploy the cf-container-broker/Docker component.

To get a head start, your homework is to review:

• the BOSH release https://github.com/cf-platform-eng/docker-boshrelease and
• BOSH workspace https://github.com/cloudfoundry-community/docker-services-boshworkspace