titaniumbunker.com

Evil geniuses and world domination are 2 of our goals... we also like Dr Who

Jenkins Build Services for making films

no comment
Screenshot from Jenkins showing the Chuck Norris plugin.

Chuck Norris is watching you build code – careful now!

For the last few days, I have been playing with the Jenkins Continuous Integration server and python, and I have reached the following conclusion: Writing python code without an effective IDE makes the job of software development harder than it needs to be.  I’ve been developing a lot on Ubuntu as well lately, so I’ve found the joy that is Wingware IDE

So – I think a bit of a recap is in order.

Recap

Set your way back machines to last year.  In fact set it to 04-05 October 2014.

So that was OggCamp.  Back then there was in interesting talk about creating animation using Free Libre and Open Source Software given by this reprobate:

Dave has been working on the SnailTales project for a while now, and with OggCamp 2015 approaching at a rate of knots I thought it was worth digging the project out, kicking the tyres and seeing how it was looking.

 

My idea for the project was to take the idea of continuous integration that is used in my day job, and apply the same ideas and technologies, but for films.  I started experimenting with the Jenkins build server last year and had got quite far – what I needed to do now was to finish the scripts.

… and make the scripts work

……… and make Jenkins put it all together.

Making tough decisions

I have also made the decision to move the scargo hosting for source from Bazaar(BZR) to Git.  This was mainly due to so much code currently being stored on GitHub.  At TitaniumBunker we use GitHub to host most if not all of the projects we have been involved in – including the recent AWTomation project, and Canonical’s recent announcement that Launchpad will now support Github for source storage, meant that we could leverage the infrastructure provided by Canonical, and also take advantage of the benefits that Github also provide.  There was also another reason.

The Jenkins support for BZR exists but doesn’t appear to be as well implemented as the support for GitHub.  BZR relied on you setting up and configuring the version control outside of Jenkins, whereas GitHub links into the Credentials store within Jenkins.  The Jenkins server can now authenticate with Github, as its own entity.  The BZR parts of the script  also relied on a number of – well there’s no nice way of saying it: but ‘fudges’ to allow the script to work correctly.

Initially the job to build scargo:

  • Obtained the  BZR revision number, and passed that into a variable for use later.
  • Pulled the latest code from the remote Launchpad BZR trunk.
  • Injected the PRE SCM step variable into the Build Job
  • Build Job got current version number – was able to then compare these numbers.
    • Are these version numbers different?  Something changed
    • Are these version number equal?  Nothing has changed – do nothing and quit.
  • Pass a collection of criteria (including the BZR_FROM / BZR_TO revision numbers) into the script (fixer.py)

The fixer script would thenNumber same?  Nothing has changed – do nothing and quit.

  • pull a list of changed files from BZR
  • iterate through each of the files that have changed.
    • Rendering any Synfig files, and then rendering the Launchpad project
    • Replace any stills with the same named Synfig rendered files.
    • Re-write the Asset reference locations
  • Create a script that will compile the video

Then a later build step

  • Runs the script that has been generated
  • Adds the newly added video files into the project.

It was at this stage that the issues with the BZR started to become apparent.  There was no real way of creating a BZR user credential that wasn’t tied into the server.  The server would also need its own PGP key, and an identity that it could use to sign into Launchpad, and commit the newly added code changes. This became a huge problem because I am running Jenkins on a virtual Server.  And I’ll be honest here.  I never realised that creating random numbers on a virtual machine would be such a problem – but it is.  Trying to generate a PGP key on a virtualized Linux box took forever.  It was so slow in fact, that many forums were advocating the use of rng-tools to try and increase the amount of system entropy available.  Purist that I am, I didn’t want to install tools on a virtual machine just to get around the fact that it is running on a virtual machine.  Therefore, I decided that maybe Git was an alternative here – it doesn’t need a certificate to authenticate to Github, can use the Jenkins credentials store, and didn’t require a lot of configuration outside Jenkins.

 

I also decided that this solution should become simpler.  I realized that the script in its current form was too tightly coupled to the version control system.  The script really only supported BZR, and moving the version control to another system – such as Git – really exposed those issues.   Versions in BZR are expressed as numbers, whereas versions in git are expressed as SHA1 checksums.  I’d also realised that making a script that could conditionally build if there are changes to the  source files meant that this script had to do 2 things well.  It had to determine whether something had changed in the project, and then build and swap out resources.  This sort of seems at odds with the UNIX philosophy of doing one thing and doing it well.  Plus Jenkins and other CI solutions already had mechanisms for detecting change within a source repository – mainly through hooks.  I ripped out many of the BZR options from the script and tried to simplify.  There are still potential areas where perhaps the script could be split into separate parts (maybe the part of the script that deals with fixing resource locations could be separated from the part that swaps out the newly built resources).

Removing the BZR version system meant that the script options were now reduced down to this:

 

Parameter Default Value Purpose
width 640 Specifies the width of the rendered animation.
height width Specifies the height of the rendered animation.
frame frame.png This is the file name used to generated individual frames from Synfig.  After a scene is rendered as video these frames are deleted.
fps 25 The Frames Per Second for the final rendered animation.  This setting is used in both the render from Synfig, and the render from kdenlive.
kdenlive storyboard.kdenlive Name of ther kdenlive file to render.  Typically this will default to storyboard.kdenlive – however it can be renamed to whatever is required.
fix False Fix is a boolean attribute.  Pass in the parameter fix (which sets that to true), and the system will only fix the resource locations.

 

Configuring the Snail Tales Build Job

The main configuration for Jenkins is performed through the Jenkins web-based front end.  The exact configuration is based mainly on what plugins are available.  For example: If you install the Chuck Norris plugin (and why wouldn’t you), then you can add a post build action, to activate Chuck Norris.

The Job configuration file for scargo

The Job configuration file for scargo

Jenkins stores the configuration for a job in the /var/lib/jenkins/job/ folder, in a config.xml file.

Config
Config
config.xml
1.7 KiB
133 Downloads
Details...

There were a number of plugins that were installed to make this possible :

List of Jenkins plugins that are installed.

Jenkins plugins that are installed on the Jenkins server

This list of installed plugins is:
Plugin Name
Version Installed
This plugin adds Apache Ant support to Jenkins
 1.2
This plugin integrates the Bazaar Version Control System with Jenkins.
1.22
ChuckNorris plugin displays a picture of Chuck Norris (instead of Jenkins the butler) and a random Chuck Norris ‘The Programmer’ fact on each build page.
0.5
This plugin allows you to store credentials in Jenkins.
1.22
Integrates Jenkins with CVS version control system using a modified version of the Netbeans cvsclient.
2.11
This plugin makes it possible to set an environment for the builds.
1.91.2
Adds the ability to monitor the result of externally executed jobs.
1.2
Shared library plugin for other Git related Jenkins plugins.
1.18.0
This plugin integrates GIT with Jenkins.
2.4.0
This plugin provides GitHub API for other plugins.
1.69
This plugin integrates GitHub to Jenkins.
1.12.0
This plugin adds Javadoc support to Jenkins.
1.1
Allows JUnit-format test results to be published.
1.0
Adds LDAP authentication to Jenkins
1.6
This plugin allows you to configure email notifications. This is a break-out of the original core based email component.
1.15
Offers matrix-based security authorization strategies (global and per-project).
1.1
Multi-configuration (matrix) project type.
1.4
Jenkins plugin for building Maven 2/3 jobs via a special project type.
2.5
Uses the OWASP Java HTML Sanitizer to allow safe-seeming HTML markup to be entered in project descriptions and the like.
1.1
Adds Unix Pluggable Authentication Module (PAM) support to Jenkins.
1.1
This plugin allows build steps to be performed before the SCM step performs an action.
0.3
This plugin provides a new enhanced API for interacting with SCM systems.
0.2
This plugin allows you to store SSH credentials in Jenkins.
1.11
This plugin allows you to manage slaves running on \*nix machines over SSH.
1.5
This plugin adds the Subversion support (via SVNKit) to Jenkins.
1.54
This plugin adds an additional dialog box in every page, which enables people to contribute localizations for the messages they are seeing in the current page.
1.10
Allows you to connect to Windows machines and start slave agents on them.
1.0
It is possible that not all of these plugins are required -in fact I will be experimenting further to determine what plugins are actually needed for this job to run.
Do you really need that plugin slide from Andrew Bayer's 2014 presentation Seven Habbits of Highliy Effective Jenkins Users"

Do you really need that plugin slide from Andrew Bayer’s 2014 presentation Seven Habits of Highly Effective Jenkins Users”

Setting up a hook into Github.

At the moment, the job is run manually.  What is needed is to change the trigger from running this job, to something that doesn’t require me logging on and pressing a button.  The process for creating such a hook is covered in this blog post and there’s even a tutorial here.  Here’s what I did to set up the hook.
  • In the Jenkins job – I clicked the option to “Build when a change is pushed to Github”
  • In the Github project – I set up the Jenkins hook URL to http://addressofserver/github-webhook/

I then pushed a version of the project to GitHub, and could see that the job was queued to start processing.

Doing something with the video

The output of the script – an MP4 file – will exist on the build server until the next build overwrites it. A better solution might be to put this somewhere where this can be stored.  Typically this can be done be creating a deployment step and taking the produced file and putting it somewhere. In some organizations, this could be a system such as Artifactory, and there is an Artifactory plugin for Jenkins – so this is a possibility, but with a video output, perhaps the best solution would be to upload this video to a video hosting platform – such as YouTube.

Now, there is no YouTube plugin to allow Jenkins output to be pushed directly to YouTube, but there are projects – such as youtube-upload that allows you to send a video via a python script – and I successfully used such a script to upload a video to the Jenkins Server’s own YouTube Chanel.

My next tasks will be to:

  • Prove that the script does indeed replace animatic frames with actual animation
  • Look at addingYouTube support to the scargo utility folder.
  • Produce some video documentation that can demo this project for OggCamp

 

Tags: , ,

Comments are closed.



Categories

Archives

Tags