As we gear up to launch SHIELD v8 at the end of 2017, it occurs to me that SHIELD is nearing its second birthday. To celebrate, I thought I’d take a look back through SHIELD two-year history.
(You may also be interested in previewing some of SHIELD v8’s forthcoming features, in SHIELD: Looking Forward)
In October 2015, a small team inside of Stark & Wayne put together a design document that would become the basis of the SHIELD implementation. We were tasked with answering one of the biggest questions facing infrastructure teams: How do I safeguard my critical data?
As a consulting firm, we’d been thinking about our customers’ data almost constantly since the beginning. The ability to spin up entire infrastructures with a single bosh deploy hasn’t helped the data protection issue — if anything it’s gotten worse. Throw into the mix agile platforms like Pivotal’s Cloud Foundry, replete with data-services brokers, and the body of critical data that needs protected only gets larger.
SHIELD was our answer to that problem.
When we started, we had a small list of data systems to backup, and a requirement that we be able to store them in S3. Due to some past experiences, the design team insisted that SHIELD be pluggable – adding support for new data systems would not require that the SHIELD software itself be rebuilt.
By early November, the core API and the inner workings of the scheduler were complete. By late November, SHIELD sported a new plugin framework that took most of the drudgery and repetition out of plugin authoring.
The SSH agent architecture was lifted from the wildly-successful Concourse CI/CD system — the SHIELD core would open up an SSH tunnel to target and storage agents whenever it ran a backup or restore job. This simple yet powerful paradigm still exists in SHIELD today, because it provides confidentiality of inflight data, message integrity (no corrupt backups!) and simple authentication.
By Thanksgiving, we deployed SHIELD to our first production environment, and began backing up several Cloud Foundry instances to Amazon S3.
By Christmas, we had a first-class SHIELD command-line utility for managing SHIELD without using raw HTTP REST API calls and curl.
Q1 of 2016 saw several weeks of internal refactoring and code cleanups. When you move as fast as the team did during the early days, you’re bound to rack up some technical debt and it’s always a good idea to go back and pare that down periodically. By this time, we had a large and growing list of plugins: PostgreSQL, MySQL, the filesystem plugin, Amazon S3, and several Cloud Foundry service broker plugins for everything from RabbitMQ to Redis.
Fun Fact: there are 14 commits against SHIELD consisting of nothing but go fmt cleanup runs.
By mid-2016, the SHIELD CLI had been re-written from the ground-up to the one you know and love today, and SHIELD’s web UI was more or less complete.
I wanted to take a moment to thank all of the people who helped SHIELD along its way. We wouldn’t be where we are today were it not for these heroic individuals who said, "Build a backup / restore system? Why not?"
- Quintessence Anx for all of her help on the SHIELD Core (database, scheduler, API)
- Thomas Mitchell for his efforts to keep the CLI elegant and functional.
- Geoff Franks for the bug finding, the bug fixing, and the entirety of the SHIELD Plugin Framework
- Dennis Bell for pulling us out of the dark ages and porting the original CLI from shell to Go.
Over the past year and a half, we’ve been busy running SHIELD and integrating with various customer data systems, all the while planning the next big round of features. As we run up to the release of SHIELD v8, we’ve got a lot of new and exciting features to show off, including Github and Cloud Foundry UAA-based authentication, at-rest encryption, and a fluid and easy-to-use web user interface.
Stay tuned!
Spread the word
