Stark & Wayne

Multi-tenant ELK for your private Cloud Foundry

If you're using a public Cloud Foundry - Pivotal Web Services, IBM Blue Mix - then you are probably ok with the idea of using a public log storage service like Papertrail or Logentries. What about if you are using your own Cloud Foundry - open source or PivotalCF - and you're not allowed to use a hosted log service? What are your options?

Our various clients have needed something and I thought I'd share one idea. It has pros and cons.

Pretty?

Events over time:

events-over-time

Line-by-line logs

line-by-line

Pros and cons

The pros are:

The cons are:

But, above all the cons are the pros - it exists and it is multi-tenant.

Perhaps the solution will work for you long enough until a better solution is proposed (hopefully by us, but we'd love any solution).

Why not logsearch?

http://www.logsearch.io/ is a distribution of ELK that include a BOSH release and BOSH workspace and serves as a great backend for Cloud Foundry component logs.

And with a firehose nozzle [see firehose-to-syslog] for loggregator it could also store all applications' logs.

But there is no multi-tenancy for Kibana - the UI portion of ELK - to restrict what a user can see. We only want a user to see the logs that a) they want to see; b) they have permission to see.

Elastic, the company, does have a product Shield that might solve this problem. I learnt about this yesterday and will investigate it later.

How do we do multi-tenant ELK?

Two components:

A platform engineer will deploy the former component once and it will be used by all users as a Cloud Foundry service.

Each application developer will deploy the latter app so they can view their logstash service instance.

This solution means that you application's logs are isolated inside a dedicated Docker container that runs logstash and elastic search.

To view the logs via Kibana you run the kibana-me-logs app as another Cloud Foundry app.

The images above are examples.

Coming soon

Currently I'm working on a way to make it very simple for platform engineers to deploy the cf-container-broker/Docker component.

To get a head start, your homework is to review: