Using Concourse CI to test your web app and PostgreSQL

A typical web application will need services, such as databases and caches, to run. It will probably also need these services to run its test suite. In this blog post, we will look at how to set up and run PostgreSQL within Concourse CI prior to running our application’s test suite.

The Problem

For a Ruby on Rails web app example, ideally you’d like a ci/scripts/tests.sh wrapper script that was simply:

#!/bin/bash
# install ruby dependencies, create databases, update schema, and run tests
bundle install
bundle exec rails test

Unfortunately we have not configured the application about its service dependencies (e.g. PostgreSQL). Nor does the container running this wrapper script actually contain a running PostgreSQL service. And that right there is the problem we will solve in this blog post.

The Solution

The following script fixes both issues – it will install & configure PostgreSQL, and it will configure the web app to use the local PostgreSQL service.

#!/bin/bash
# assumes this script is in ci/scripts/ folder
# so change to root directory of app
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd $DIR/../..
# fail fast if any command errors
set -e
# install the postgresql service + client libraries
apt-get update
apt-get install -y postgresql libpq-dev
# allow access from any local client as `postgres` user
cat > /etc/postgresql/*/main/pg_hba.conf <<-EOF
local   all   postgres   trust
EOF
# start the postgres service
service postgresql restart
#
# This point onwards is specific to Ruby on Rails
#
# configure the application to use the local DB
cat > config/database.yml <<-YAML
default: &default
  adapter: postgresql
  encoding: unicode
  pool: 5
  username: postgres
  template: template0
development:
  <<: *default
  database: collective-training-app_development
test:
  <<: *default
  database: collective-training-app_test
production:
  <<: *default
  database: collective-training-app_production
YAML
# install ruby dependencies, create databases, update schema, and run tests
bundle install
bundle exec rails db:create:all
bundle exec rails db:migrate
bundle exec rails test

The Speed Up

This script takes about two minutes to run for an application with a small rails test suite. The breakdown of time:

• installing and configuring PostgreSQL – 13 seconds
• downloading and installing Ruby gems (bundle install) – 85 seconds
• running PostgreSQL, creating databases and schemas – 8 seconds
• running tests – 7 seconds

Since the version of PostgreSQL and most of the web application dependencies (RubyGems in the example above) are relatively static, you could gain a huge percentage speed improvement by pre-installing PostgresSQL and the RubyGems into the Docker image you are using.

In my pipeline, the test task dropped to 23 seconds instead of approximately 120 seconds.

Fortunately Concourse CI is also very good at creating and updating Docker images for use by other Concourse CI tasks. See our Concourse Tutorial lesson on Creating and Using Docker Images.