Mar 12, 2015 Multi-tenant ELK for your private Cloud Foundry
If you're using a public Cloud Foundry - Pivotal Web Services, IBM Blue Mix - then you are probably ok with the idea of using a public log storage service like Papertrail or Logentries. What about if you are using your own Cloud Foundry - open source or PivotalCF - and you're not allowed to use a hosted log service? What are your options?
Our various clients have needed something and I thought I'd share one idea. It has pros and cons.
Events over time:
Pros and cons
The pros are:
- using well known Elastic Search/Logstash/Kibana stack [ELK]
- it exists as a solution
- it allows multi-tenancy isolation of logs
- easy deployment via bosh-workspace for deploying docker/logstash as a service
- all components are open source
The cons are:
- it uses the relatively unmaintained cf-containers-broker project
- not easily horizontally scalable - the
cf-containers-brokerproject only supports a single Docker server
- each logstash service instance is running in a single Docker container and cannot be scaled out
- limited ops tooling to manage the logstash docker containers once they are instantiated
- the kibana app includes an insecure proxy through to elastic search
But, above all the cons are the pros - it exists and it is multi-tenant.
Perhaps the solution will work for you long enough until a better solution is proposed (hopefully by us, but we'd love any solution).
Why not logsearch?
And with a firehose nozzle [see firehose-to-syslog] for loggregator it could also store all applications' logs.
But there is no multi-tenancy for Kibana - the UI portion of ELK - to restrict what a user can see. We only want a user to see the logs that a) they want to see; b) they have permission to see.
Elastic, the company, does have a product Shield that might solve this problem. I learnt about this yesterday and will investigate it later.
How do we do multi-tenant ELK?
- cf-containers-broker is a Cloud Foundry service broker that locally provisions Docker containers - we can use it to run ELK in a Docker container
- kibana-me-logs is an app that shows the Kibana UI to display your application's logs
A platform engineer will deploy the former component once and it will be used by all users as a Cloud Foundry service.
Each application developer will deploy the latter app so they can view their logstash service instance.
This solution means that you application's logs are isolated inside a dedicated Docker container that runs logstash and elastic search.
To view the logs via Kibana you run the
kibana-me-logs app as another Cloud Foundry app.
The images above are examples.
Currently I'm working on a way to make it very simple for platform engineers to deploy the
To get a head start, your homework is to review:
- the BOSH release https://github.com/cf-platform-eng/docker-boshrelease and
- BOSH workspace https://github.com/cloudfoundry-community/docker-services-boshworkspace