Easy to use developer platforms, such as Cloud Foundry, make it more and more desirable to build systems using microservices, rather than monolithic apps. Each component can be scaled independently, each component can be tested and introduced independently, and as a bonus it can be desirable for engineering groups to organize around microservice components. I was asked, "how do we protect our backend microservices from public access?"
Good question. Here’s a picture to help with the answer below:
@drnic @cloudfoundry thx Dr Nic exactly what we are doing. Your article saves me some documentation.
— René Welches (@rwelches) October 31, 2014
The question of protecting some applications from external access assumes that Cloud Foundry itself is being run within private networking. That is, the right-hand side of the diagram.
For (normal) public access, a load balancer is run inside a different DMZ network that allows inbound HTTP traffic on ports 80 (http) and 443 (https). Do not allow any TCP traffic proxying such as dedicated web sockets port 4443 for reasons discussed below. That is, configured loggregator for HTTPS 443.
For private-only access, another load balancer is run inside the private network.
That’s part one of the solution: two load balancers sending their traffic to the same set of routers into the same set of runners/DEAs.
Part two is the answer to the question: how do you ensure that private-only apps cannot be accessed by the stateless load balancer in the DMZ?
The answer is: register distinct Cloud Foundry DNS domains. In the diagram above, these are apps.mypaas.com and private.mypaas.com.
In the diagram example there is a registered route myapp.apps.mypaas.com that is mapped to one application, and another route myservice.private.mypaas.com that is mapped to another application.
How do we ensure that myservice.private.mypaas.com is not valid for the public load balancer?
Setup the wildcard DNS *.apps.mypaas.com and *.private.mypaas.com to point to the two different load balancers.
Protecting the public load balancer
Finally, blacklist the public load balancer from allowing any private domain requests.
The goal here is to protect the load balancer from a 3rd party setting up their own DNS entry for *.private.mypaas.com (i.e. DNS spoofing) to the public load balancer to trick it into letting traffic flow through to the router.
The corollary of this requirement is that you must disable any TCP proxying through the public load balancer – it is an attack vector to bypass the HTTP blacklist rules you setup. You must enforce the use of HTTPS for loggregator web sockets.
What am I talking about? If you use the haproxy job template from cf-release, it opens port 4443 for pure TCP traffic. This would mean that any HTTP traffic sent at port 4443 would slip through the haproxy load balancer into the router. The router would then happily route the HTTP request.
TCP traffic through the load balancer cannot be blacklisted by URL. Only HTTP traffic (layer 7 mode containing headers). So no TCP traffic can be allowed through at all.
For additional security, all private applications should require basic auth credentials (at a minimum) for valid clients to communicate.
Summary
- separate load balancers for public vs private access
- external DNS and internal domains to allow creation of public vs private routes
- Only HTTP(S) traffic through public load balancer
- Block any private domain traffic through public load balancer
- backend services to require some authentication credentials from client apps
Feedback
If you have any extensions or concerns or can see holes in this model, please let me know. Security is hard! We’d love feedback or suggestions for the safety and success of all platform implementations!
Thanks to Dave Bishop & Long Nguyen for setting me straight on a few issues (servers cannot filter traffic via SSL as clients can just ignore invalid certs) and helping isolate attack vectors (TCP 4443).
We’re still implementing & verifying the approach. But wanted to share asap in case anyone had an alternate/better approach.
Implementation details might make a good post too. Let us know if you’re interested in these details.
Spread the word
