Faster CI - how we reduced our build time by 25%

We’ve been using Travis CI to run tests and automate deployments on our projects for the last few years and have been very happy with the service they provide. Our data visualisation tool, Dataseed, has used Travis from the start and over the years we’ve built up a large amount of tests for the font-end and back-end code. Having a comprehensive test suite is great for catching regressions and doing refactors but the length of time the build and tests were taking had become a real pain point.

In this post I’m going to look at the steps we took to reduce the time it took for the whole process to run, so that changes became visible on our staging/UAT site quicker.

Status Quo

Our .travis.yml initially looked like this:

language: python
python: 2.7

env:
    global:
        - ES_URL=https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.3.deb

        - HEROKU_BRANCH=dev
        - HEROKU_REPO=git@heroku.com:dataseed-stage.git

        - secure: <HEROKU DEPLOYMENT KEY>

services:
    # Start redis as Travis service
    - redis-server

install:
    # Install ElasticSearch
    - sudo apt-get purge elasticsearch
    - wget --output-document=/tmp/elasticsearch.deb $ES_URL
    - sudo dpkg --install /tmp/elasticsearch.deb
    - sudo service elasticsearch start

    # Install Python dependencies
    - sudo apt-get -qq install python-dev libpq-dev libevent-dev libxml2-dev libxslt-dev
    - pip install --requirement=$TRAVIS_BUILD_DIR/requirements.txt

    # Install JS dependencies
    - npm install
    - bower install

script:
    # Run Python tests
    - python $TRAVIS_BUILD_DIR/test.py

    # Run JS tests
    - gulp test

after_success:
    # Submit code coverage information to Coveralls
    - coveralls

    # Only deploy from $HEROKU_BRANCH
    # When Travis builds a pull request $TRAVIS_BRANCH is set to the base branch
    # (the branch we want to merge into). The second condition below prevents
    # from deploying pull requests with base branch dev.
    - if [ $TRAVIS_BRANCH != $HEROKU_BRANCH ] || [ $TRAVIS_PULL_REQUEST != false ]; then exit 0; fi

    # Install Heroku gem
    - wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh

    # Configure Git/SSH
    - git remote add heroku $HEROKU_REPO
    - git fetch --unshallow
    - echo "Host heroku.com" >> ~/.ssh/config
    - echo "  StrictHostKeyChecking no" >> ~/.ssh/config
    - echo "  CheckHostIP no" >> ~/.ssh/config
    - echo "  UserKnownHostsFile=/dev/null" >> ~/.ssh/config

    # Add Heroku SSH keys
    - heroku keys:clear
    - yes | heroku keys:add

    # Deploy to Heroku (always "master" branch)
    - yes | git push --force heroku $TRAVIS_BRANCH:master

As you can see, the steps that are taken are:

  1. Install a specific version of ElasticSearch
  2. Install Python dependencies
  3. Install Javascript dependencies
  4. Run back-end tests
  5. Run front-end tests
  6. Deploy to Heroku

Below I’ll go through the various changes we made and the rationale behind each one.

APT Packages

In order to use Travis’s container-based infrastructure we needed to eliminate all uses of sudo. As you can see above, it was only used for installing additional packages with apt which we were able to replace with the Travis apt add-on:

sudo: false

cache:
    apt: true

addons:
    apt:
        sources:
            - elasticsearch-1.7
        packages:
            - elasticsearch
            - libpq-dev
            - libevent-dev
            - libxml2-dev
            - libxslt1-dev
            - python-dev

Note that we had to add the elasticsearch-1.7 source so that we could use a specific version (1.7) of ElasticSearch. You can see what non-default packages are available, and request new ones, in the apt-source-whitelist Github repository.

By switching to containers we gained the following:

Javascript Dependencies

We used Travis’s caching to cache downloads of packages from npm and Bower. Unfortunately this only saved us a couple of seconds as most of the time spent by these package managers is moving the packages into place.

cache:
    directories:
        - $HOME/.cache/bower
        - $HOME/.npm

Python Dependencies

Aside from the tests themselves, the slowest part of the process was installing the Python dependencies from PyPI. By using Python wheels in combination with Travis’s caching we managed to reduce the median time for this step from 4 minutes to 15 seconds!

cache:
    directories:
        - $HOME/.cache/pip

install:
    - pip install -U pip wheel
    - pip install --requirement=$TRAVIS_BUILD_DIR/requirements.txt

Back-End Tests

As you’d expect, the slowest part of the process has always been running the tests themselves. The simplest way to speed the tests up was to run them in parallel, which we used the nose multiprocess plugin for. The test.py script used to run the test suite was updated as follows:

from nose import run
from nose.plugins.multiprocess import MultiProcess

run(argv=[
        'tests',
        '--processes=2',
        '--with-coverage',
        '--cover-package=dataseed',
    ])
    plugins=[MultiProcess()])

Heroku Deploy

When we first wrote our Travis configuration integrated Heroku deployment wasn’t available from Travis. Now that this is provided by Travis we can remove our custom code to deploy via Git/SSH and in the process save a few seconds.

deploy:
    provider: heroku
    app: <HEROKU APP NAME>
    on: dev
    api_key:
        secure: <HEROKU API TOKEN>

This simply deploys to our Heroku staging app when commits are made to the dev branch (and all tests have passed). To add the encrypted API key we used the following command:

travis encrypt $(heroku auth:token) --add deploy.api_key

Conclusion

After putting everything together, our final configuration was as follows:

language: python
python: 2.7

sudo: false

cache:
    apt: true
    directories:
        - $HOME/.cache/pip
        - $HOME/.cache/bower
        - $HOME/.npm

addons:
    apt:
        sources:
            - elasticsearch-1.7
        packages:
            - elasticsearch
            - libpq-dev
            - libevent-dev
            - libxml2-dev
            - libxslt1-dev
            - python-dev

services:
    # Start Redis and ElasticSearch as Travis service
    - redis-server
    - elasticsearch

install:
    # Install Python dependencies
    - pip install -U pip wheel
    - pip install --requirement=$TRAVIS_BUILD_DIR/requirements.txt

    # Install JS dependencies
    - npm install
    - bower install

script:
    # Run Python tests
    - python $TRAVIS_BUILD_DIR/test.py

    # Run JS tests
    - gulp test

after_success:
    # Submit code coverage information to Coveralls
    - coveralls

deploy:
    provider: heroku
    app: <HEROKU APP NAME>
    on: dev
    api_key:
        secure: <HEROKU API TOKEN>

In total we reduced the time from commit to deploy by 25%, which was much appreciated by developers and QA! Reducing friction in this way makes for a quicker development and testing cycle which ultimately gets new functionality into the hands of our users faster than before.