titaniumbunker.com

Evil geniuses and world domination are 2 of our goals... we also like Dr Who

Making the magic work

no comment

You’re a wizard, Harry… if only you could just get continuous integration to work

I’ve been focusing my efforts on the selfietorium lately and in particular how to combine all the various support systems with GitHub, where the source is stored. This blog post details making the magic work: to get Continuous Integration, build, package and release uploads working.

Continuous Integration is the foundation from which the other support services will hang.  There’s no point in performing code analysis on code that doesn’t build or pass its tests.  So, let’s get started.

In previous projects – like Snail Tales – we have created Jenkins installs and created build scripts for all of this to work.  For the selfietorium project we are using Travis-CI.

Selfietorium is a raspberry pi based Python project, and there is great support in Travis-CI for Python – some languages such as c# are not 100% supported, so Travis-CI may not be suitable for all uses.  Before you start looking at using Travis-CI for your solution, you should probably check that the language is supported by taking a look at the getting started page in the Travis-CI Docs.

Techies amongst you might be thinking

Mike – what are you going to build?  Python is an interpreted language – there is no compiler for Python

And that’s true enough.  I aim to use the Travis-CI build system to run my unit tests (when I write some) and package my Python code into a Debian .deb file to allow easy installation onto a Raspberry Pi.

So let’s get cracking

To start with, you’ll need an account on Travis-CI.  Travis-CI uses GitHub for authentication, so that’s not too difficult to set up – just sign in with your GitHub account.

Now that you have an account what do you do next?  There are a couple of things you need to do to make your project build:  Create your project within Travis-CI and create a .travis.yml file

The .travis.yml file contains all of the steps to build and process your project, and it can be somewhat complicated.  What is amazingly simple though is setting up a GitHub repository to build.  Travis-CI presents me with all of the repositories that are capable of being built.  From here I picked the TitaniumBunker/Selfietorium repository, and that was pretty much it.

Picking which repository to build is probably the simplest part of this set up.

Once your repository is set up it needs to be configured – the docs are an absolute must here. There is no IDE to manage your configuration – all that stands between build success and multiple frustrating build failures is you and your ability to write a decent .travis.yml file.

Nothing will build until you next push something to your GitHub repository.  Push something to your repository and Travis-CI will spring into life, and potentially fail with an error, probably looking something like this:

Worker information
hostname: ip-10-12-2-57:94955ffd-d111-46f9-ae1e-934bb94a5b20
version: v2.5.0-8-g19ea9c2 https://github.com/travis-ci/worker/tree/19ea9c20425c78100500c7cc935892b47024922c
instance: ad8e75d:travis:ruby
startup: 653.84368ms
Could not find .travis.yml, using standard configuration.
Build system information
Build language: ruby
Build group: stable
Build dist: precise
Build id: 185930222
Job id: 185930223
travis-build version: 7cac7d393
Build image provisioning date and time
Thu Feb  5 15:09:33 UTC 2015
Operating System Details
Distributor ID:	Ubuntu
Description:	Ubuntu 12.04.5 LTS
Release:	12.04
Codename:	precise
...
...
...

There’s a lot of cruft in there – but the lines that are interesting are:

  • version – The version line hints that the Travis-CI worker code is on GitHub.  It is.
  • Could not find .travis.yml, using standard configuration. – The build fails to find a .travis.yml file and defaults to building a Ruby project.
  • Description: Ubuntu 12.04.5 LTS – the build workers seem to be Ubuntu based.
  • Cookbooks Version a68419e – Travis cookbooks are used with chef to set up workers

.travis.yml file

The .travis.yml file is effectively a script that executes as the build life cycle executes. The Customizing the Build page says that a build is made up of 2 main steps

  1. install: install any dependencies required
  2. script: run the build script

These 2 sections are essential for any .travis.yml file. There can be more than just these 2 sections, and the Customizing the Build page details a whole bunch of extra steps that can be added to your .travis.yml file.

The. travis.yml file for selfietorium looks like this:

dist: trusty
sudo: required

addons:
  sonarqube:
    token:
      secure: '$SONARQUBE_API_KEY'


language: python

python:
 - "2.7"
# uncomment and edit the following line if your project needs to run something other than `rake`:

before_install:
  - sudo apt-get -qq update
  - sudo apt-get install -y build-essential devscripts ubuntu-dev-tools debhelper dh-make diffutils patch cdbs
  - sudo apt-get install -y dh-python python-all python-setuptools python3-all python3-setuptools
  - sudo apt-get install -y python-cairo python-lxml python-rsvg python-twitter
  
install: true
 
script:
   - sonar-scanner
   - sudo dpkg-buildpackage -us -uc 

before_deploy:
  cp ../python-selfietorium_1_all.deb python-selfietorium_1_all.deb
tyle="white-space: pre-wrap; word-wrap: break-word;"
deploy:
  provider: releases
  api_key: '$GITHUB_API_KEY'
  file: 'python-selfietorium_1_all.deb'
  skip_cleanup: true
  on:
    branch: master
    tags: true

So let’s break down what this script does.

dist: trusty
sudo: required

addons:
  sonarqube:
    token:
      secure: '$SONARQUBE_API_KEY'


language: python

python:
 - "2.7"

This section is the pre-requisites section. It tells Travis-CI that the worker that is going to run this script should be an Ubuntu 14.04 LTS (Trusty Tahr) based machine.  Travis-CI will build on either a Virtual Machine environment (with sudo enabled), or a Container – which is I believe based on Docker.  The issue with docker is that while it takes seconds to provision a container based environment, it currently doesn’t have sudo available to it, meaning that performing activities using sudo (for example, installing build dependencies) is not possible in a container based environment.  The Travis blog does state that:

If you require sudo, for instance to install Ubuntu packages, a workaround is to use precompiled binaries, uploading them to S3 and downloading them as part of your build, installing them into a non-root directory.

Now I still have some work to do around dependency resolution – I think it is possible to trim the number of dependencies right down. At the moment the build system installs all of the runtime dependencies which might potentially be overkill for the packaging – however they still might be needed for unit testing. Further work is required to look into that. If these dependencies can be removed, then the build could potentially be done in a container, speeding up the whole process. I can almost hear the other fellow techies…

But Mike, why don’t you use Python distutils, and use pypi to install your dependencies?

A fair question.  Using pypi would mean that I could potentially install the dependencies without needing sudo access – the issue is that python-rsvg doesn’t seem to be available on pypi, and only seems to be available as a Linux package.

In this section I’m also telling Travis-CI that I would like to use SonarQube to perform analysis on the solution [1] and that the solution language is Python 2.7.  I think the general opinion of developers out there is:

Use Python 3 – because Python 2 is the old way, and Python 3 is the newest

I’d like to use the new and shiny Python 3, but I fear that there may be libraries that I am using that have no Python 3 implementation – and that fear has led me back into the warm embrace that is Python 2.7.  I plan to perform an audit and determine whether the project can be ported to Python 3.

before_install:
  - sudo apt-get -qq update
  - sudo apt-get install -y build-essential devscripts ubuntu-dev-tools debhelper dh-make diffutils patch cdbs
  - sudo apt-get install -y dh-python python-all python-setuptools python3-all python3-setuptools
  - sudo apt-get install -y python-cairo python-lxml python-rsvg python-twitter

In this section I am installing the various Linux packages required to perform the build.  These are standard commands for installing packages onto an environment.

install: true

And here the installation does nothing.  As I said at the top of this article, there is no build for Python programs.

script:
   - sonar-scanner
   - sudo dpkg-buildpackage -us -uc

Right about here is where I would be running any unit tests – but I don’t have any yet.  This script then sends the code to SonarQube – a topic for a future post – and then calls dpkg-buildpackage to create the binary package.  At the end of this step we have a deb file that could potentially be deployed.

before_deploy:
  cp ../python-selfietorium_1_all.deb python-selfietorium_1_all.deb

Before I deploy the deb file, I need to copy it, so I copy the generated deb file into the current working directory.

deploy:
  provider: releases
  api_key: '$GITHUB_API_KEY'
  file: 'python-selfietorium_1_all.deb'
  skip_cleanup: true
  on:
    branch: master
    tags: true

Finally, we deploy the file.  The provider: releases line tells Travis-CI to use the GitHub Releases provider to push the build artifact to a GitHub release.

It uses a secret API key to gain access to the project releases.  The file is the name of the generated file, and the skip_cleanup prevents Travis-CI from resetting the working directory and deleting all changes made during the build.  The on section controls when files are deployed.  With this setting, only changes made to the master branch, that are tags trigger the deployment.  GitHub releases are actually tags, so creating a release creates a tag on a branch.  For selfietorium we create releases on the master branch.  The release deployment then pushes the build artifact to GitHub effectively offering it as the binary file for that release tag – and uploading it to the GitHub Release.

Keeping Secrets.

In order for Travis-CI to upload your build artifact, it will need to identify itself, and to do that we create a Personal Access Token.  Using this Token, the GitHub Releases provider can now communicate with GitHub as if it was you.  We can’t just add the GitHub token to our .travis.yml file.  Well, I suppose we could, but then we shouldn’t be surprised if other files start appearing in our releases.  The .travis.yml file is publicly available on GitHub – so we need a way of storing the token securely, and injecting it into the script during execution.  Travis-CI offers an ability to store environment variables within a project.  These variables are hidden when the log files are produced if you clear the display value in build log check box.  To use that variable in your .travis.yml file you’d refer to it like this:

$GITHUB_API_KEY

Travis-CI Environment Variables

Grabbing a Status badge

Within the Travis-CI project settings screen, clicking the status badge offers the ability to generate suitable markdown for GitHub.

Adding a spiffy status badge to your GitHub ReadMe.md markdown could not be easier

Quick Recap.

So what we’ve done so far is:

  • Configured Travis-CI to build when we push to the repository.
    • Eventually this will allow for unit tests to be run – but at the moment there are no unit tests for selfietorium.
  • Configured Travis-CI to package the application into a .deb file when a release is created.
    • Releases are effectively tags within a git repository.
  • Configured Travis-CI to deploy out build artifact back to GitHub using a Personal Access Token.
    • Personal Access Token is securely stored in a Travis-CI environment variable.
  • We’ve created some spiffy markdown for a status badge that we can incorporate into our repository markdown.

Debian file built using Travis-CI and deployed to GitHub

Here’s what it looks like when you run the installer under Ubuntu:

Selfietorium installation through Ubuntu Software Centre

In a forthcoming as yet un-written post, I’ll document how to set up the packaging folder so that selfietorium is installable and executable on a Linux system.  It will probably borrow quite heavily from this page I wrote about content packaging.

Useful tools

Travis WebLint – check your Travis configuration for obvious errors.

Footnotes

  • [1] I’ll return to that in a later post when I’ll talk about Continuous Inspection

Categories

Archives

Tags