May 23, 2019 Part 2: Installing Cloud Foundry Sidecars with Buildpacks
Sidecars allow additional processes that share filesystem and socks with a Cloud Foundry application. We investigate separating these concerns from app developers by distributing and installing them with Buildpacks.
This article is almost Part 2 in a 5-part series. Part 1 starts with Tim Downey's How to Push an App to Cloud Foundry with Sidecars. In this article, Part 2, I demonstrate the idea of distributing a Sidecar within a Buildpack. In the next article, Part 3, we will finish the job of packaging Tim's example Sidecar into a Buildpack. In Part 4, we will refactor the Buildpack to include cached assets so it can be installed into many Cloud Foundry foundations for all users to use. Finally, in Part 5 we look at integration testing our buildpack within applications running on Cloud Foundry to ensure they work.
There is an incredible reason to upgrade to latest open source Cloud Foundry, and to migrate more applications to your Cloud Foundry foundations: sidecars.
Sidecars in Cloud Foundry are additional dependent processes that are run in the same container as the main application process.
With sidecars you can now run APM agents, secure configuration services, proxies, and more within the same container, filesystem, and localhost networking as the main application process.
How to Push an App to Cloud Foundry with Sidecars was written by the Cloud Foundry CAPI team's Tim Downey. He outlines some use cases for sidecars that has Cloud Foundry fans very excited:
You may want to use sidecar processes if you have two processes that:
- need to communicate over a unix socket or via localhost.
- need to share the same filesystem.
- need to be scaled and placed together.
- need to have fast interprocess communication.
Each sidecar process is run within the same application container. Tim's architecture diagram showed that the sidecar can be authored and packaged differently to the main application:
We now catapult onwards from Tim's article.
Tim showed how an application's manifest can declare one or more sidecars. You document the command to run. You execute some software that is packaged in the droplet.
In Tim's Part 1 article, the dummy
./config-server sidecar was uploaded into the droplet from the developer's filesystem with
This method requires the application developer to correctly selected or build the correct
config-server binary to be uploaded.
Yet it is likely that the
config-server executable is authored and maintained by a different team than the application team. Once a new version is authored, it should be rolled out to all applications.
In Cloud Foundry we have a first-class method for separating these concerns.
- Sidecar developers distribute their executables with Buildpacks.
- Application developers use
cf v3-pushto upload their source code and to blend in the Sidecars with Buildpacks.
Distributing Sidecar software as a Buildpack ensures that all applications will continuously upgrade to new Sidecar versions each time they are deployed or restaged.
The rest of this article will start to refactor Tim's idea of a
./config-server sidecar into a buildpack. The subsequent article, Part 3, will finish the job of porting the sample
./config-server into a complete buildpack. In Part 4 will walk through a groundup rewrite of the buildpack to allow installation as a system-wide buildpack, including locally cached assets. In Part 5 we will author integration tests to confirm the buildpack can be installed, and works correctly when deployed with an application.
Upgrade Cloud Foundry CAPI
Sidecar support was initially introduced in CAPI version 1.79.0. You need to bump your CAPI release, or wholesale upgrade your Cloud Foundry to include at least this CAPI version.
I will use CFDev, which added sidecar support yesterday with 0.0.16 release.
cf install-plugin -r CF-Community cfdev cf dev start
In this article we will create a new Buildpack that installs a
config-server executable during application staging. But it will be a silly piece of software that does nothing. Yet our silly nothing software will be installed into the droplet will be executed as a sidecar. In the Part 3 article we will replace the silly software and the buildpack will bundle a proper Golang application with which our application can interact.
The buildpack for this article is at https://github.com/starkandwayne/part2-sidecar-buildpack.
We can see that it is a Buildpack because it contains a
This script is run during application staging and will create a
./config-server file, and make it executable, within the application's own project folder within the droplet.
Look closely and you'll see that its not a fancy pre-compiled binary. It's a silly shell script that does nothing forever.
This is a specific type of Buildpack called a Supply Buildpack. It cannot be used to run applications (like the pre-installed System Buildpacks that you commonly use each day). Instead, it can be used in addition to normal Buildpacks to install additional software. In our case, we have a Supply Buildpack to install a
config-server executable that will be invoked as a sidecar.
Our Buildpack repository also includes a dev/test/demo application. Actually, its the same as Tim's sample application. Your buildpacks will want to include one or more dev/test/sample applications to allow you to work on your buildpack and ensure it works within normal Cloud Foundry applications as expected. Writing test suites is a good idea too, but outside the scope of this article.
fixtures/rubyapp for the sample app I borrowed from Tim.
The most important part is
manifest.yml which describes to Cloud Foundry that you want a sidecar included in your application container, and that the sidecar comes from an additional buildpack:
applications: - name: app-using-config-server env: CONFIG_SERVER_PORT: 8082 buildpacks: - https://github.com/starkandwayne/part2-sidecar-buildpack - ruby_buildpack sidecars: - name: config-server process_types: - web command: './config-server'
buildpacks: section lists our https://github.com/starkandwayne/part2-sidecar-buildpack first, and the application's buildpack (
ruby_buildpack in our example) must be last in the list (to indicated to Cloud Foundry that it is a Ruby app, and all other buildpacks are Supply-only).
sidecars: section describes one or more sidecar processes to be run inside the same container as the application process. The
name: config-server is arbitrary, although it will be used in the streaming logs to identify stdout/stderr emitting from our sidecar.
command: './config-server' describes the full command to run to launch the sidecar. In our example we assume that the
config-server sidecar file will be available in the root of the application folder. Our buildpack's
bin/supply takes responsibility for putting it there.
Whilst our silly "do nothing forever"
config-server sidecar does not yet support incoming requests for the app, we can see an environment variable
CONFIG_SERVER_PORT being set so that in future both the application and the sidecar know what localhost port to communicate over. Alternately we might use a socket or a folder of shared files.
You can see this in action with our sample1 buildpack and its demo ruby app.
If you're just bootstrapped and upgraded CFDev, first login as a regular user:
cf login -a https://api.dev.cfdev.sh --skip-ssl-validation -u user -p pass
Next, clone the buildpack project to get its demo app:
git clone https://github.com/starkandwayne/part2-sidecar-buildpack cd part2-sidecar-buildpack cf v3-create-app app-using-config-server cf v3-apply-manifest -f fixtures/rubyapp/manifest.yml cf v3-push app-using-config-server -p fixtures/rubyapp
During the staging process we can see our supply buildpack being installed:
-----> Installing config-server buildpack $BUILD_DIR: /tmp/app $CACHE_DIR: /tmp/cache/f350c7a8c3840774df22932d952d419d $ENV_DIR: /tmp/contents224840018/deps
I'm displaying the three values only for debugging/learning purposes. You wouldn't include this in your own buildpack.
We can check that our sidecar is installed and running in a couple ways.
First, it emits a message to STDOUT during its start up. We can see that the sidecar is launched before our main application:
$ cf logs app-using-config-server --recent ... [APP/PROC/WEB/SIDECAR/CONFIG-SERVER/0] OUT Starting dummy config-server... [APP/PROC/WEB/0] ERR [2019-05-18 02:53:35] INFO WEBrick 1.3.1 [APP/PROC/WEB/0] ERR [2019-05-18 02:53:35] INFO ruby 2.4.6 (2019-04-01) [x86_64-linux] [APP/PROC/WEB/0] ERR [2019-05-18 02:53:35] INFO WEBrick::HTTPServer#start: pid=16 port=8080
Second, we can SSH into the application container.
$ cf ssh app-using-config-server
Each time we SSH into an app container, run the following command to setup our environment and change to the application's root directory:
We can now see that
./config-server has been added to our application by our buildpack:
$ cat ./config-server #!/bin/bash echo "Starting dummy config-server..." tail -f /dev/null
Finally, we can see that
./config-server is a running process:
$ ps axwf PID COMMAND 17 /home/vcap/deps/1/vendor_bundle/ruby/2.4.0/bin/rackup config.ru -p 8080 7 /bin/bash ./config-server 81 \_ tail -f /dev/null ...
Sidecars within our application containers is an important architecture paradigm and its fantastic we now have it in Cloud Foundry. I feel strongly that their development and distribution should be separated out from application development through Buildpacks.
In the next article, Part 3, we will replace our silly shell script with a precompiled Golang application that will be installed into the application droplet during staging. Our sample Ruby application will merrily communicate with its sidecar.
Thanks to Tim Downey for allowing me to consider my articles as a continuation of his original article. His article, and the CAPI team's work on Sidecars, was entirely completed before I started this article and was my own starting place for education.
Thanks to Dr. Max for organizing the Cloud Foundry CAB calls, and for Carlo Gutierrez and Sophia Turol for taking the time to write up what happens during them. During the May CAB call the CAPI team shared their work on Sidecars, and the blog post made me aware of them. You can read about previous CAB calls or watch them from the YouTube playlist.