For the last few days, I have been playing with the Jenkins Continuous Integration server and python, and I have reached the following conclusion: Writing python code without an effective IDE makes the job of software development harder than it needs to be. I’ve been developing a lot on Ubuntu as well lately, so I’ve found the joy that is Wingware IDE
So – I think a bit of a recap is in order.
Set your way back machines to last year. In fact set it to 04-05 October 2014.
So that was OggCamp. Back then there was in interesting talk about creating animation using Free Libre and Open Source Software given by this reprobate:
Dave has been working on the SnailTales project for a while now, and with OggCamp 2015 approaching at a rate of knots I thought it was worth digging the project out, kicking the tyres and seeing how it was looking.
My idea for the project was to take the idea of continuous integration that is used in my day job, and apply the same ideas and technologies, but for films. I started experimenting with the Jenkins build server last year and had got quite far – what I needed to do now was to finish the scripts.
… and make the scripts work
……… and make Jenkins put it all together.
Making tough decisions
I have also made the decision to move the scargo hosting for source from Bazaar(BZR) to Git. This was mainly due to so much code currently being stored on GitHub. At TitaniumBunker we use GitHub to host most if not all of the projects we have been involved in – including the recent AWTomation project, and Canonical’s recent announcement that Launchpad will now support Github for source storage, meant that we could leverage the infrastructure provided by Canonical, and also take advantage of the benefits that Github also provide. There was also another reason.
The Jenkins support for BZR exists but doesn’t appear to be as well implemented as the support for GitHub. BZR relied on you setting up and configuring the version control outside of Jenkins, whereas GitHub links into the Credentials store within Jenkins. The Jenkins server can now authenticate with Github, as its own entity. The BZR parts of the script also relied on a number of – well there’s no nice way of saying it: but ‘fudges’ to allow the script to work correctly.
Initially the job to build scargo:
- Obtained the BZR revision number, and passed that into a variable for use later.
- Pulled the latest code from the remote Launchpad BZR trunk.
- Injected the PRE SCM step variable into the Build Job
- Build Job got current version number – was able to then compare these numbers.
- Are these version numbers different? Something changed
- Are these version number equal? Nothing has changed – do nothing and quit.
- Pass a collection of criteria (including the BZR_FROM / BZR_TO revision numbers) into the script (fixer.py)
The fixer script would thenNumber same? Nothing has changed – do nothing and quit.
- pull a list of changed files from BZR
- iterate through each of the files that have changed.
- Rendering any Synfig files, and then rendering the Launchpad project
- Replace any stills with the same named Synfig rendered files.
- Re-write the Asset reference locations
- Create a script that will compile the video
Then a later build step
- Runs the script that has been generated
- Adds the newly added video files into the project.
It was at this stage that the issues with the BZR started to become apparent. There was no real way of creating a BZR user credential that wasn’t tied into the server. The server would also need its own PGP key, and an identity that it could use to sign into Launchpad, and commit the newly added code changes. This became a huge problem because I am running Jenkins on a virtual Server. And I’ll be honest here. I never realised that creating random numbers on a virtual machine would be such a problem – but it is. Trying to generate a PGP key on a virtualized Linux box took forever. It was so slow in fact, that many forums were advocating the use of rng-tools to try and increase the amount of system entropy available. Purist that I am, I didn’t want to install tools on a virtual machine just to get around the fact that it is running on a virtual machine. Therefore, I decided that maybe Git was an alternative here – it doesn’t need a certificate to authenticate to Github, can use the Jenkins credentials store, and didn’t require a lot of configuration outside Jenkins.
I also decided that this solution should become simpler. I realized that the script in its current form was too tightly coupled to the version control system. The script really only supported BZR, and moving the version control to another system – such as Git – really exposed those issues. Versions in BZR are expressed as numbers, whereas versions in git are expressed as SHA1 checksums. I’d also realised that making a script that could conditionally build if there are changes to the source files meant that this script had to do 2 things well. It had to determine whether something had changed in the project, and then build and swap out resources. This sort of seems at odds with the UNIX philosophy of doing one thing and doing it well. Plus Jenkins and other CI solutions already had mechanisms for detecting change within a source repository – mainly through hooks. I ripped out many of the BZR options from the script and tried to simplify. There are still potential areas where perhaps the script could be split into separate parts (maybe the part of the script that deals with fixing resource locations could be separated from the part that swaps out the newly built resources).
Removing the BZR version system meant that the script options were now reduced down to this:
|width||640||Specifies the width of the rendered animation.|
|height||width||Specifies the height of the rendered animation.|
|frame||frame.png||This is the file name used to generated individual frames from Synfig. After a scene is rendered as video these frames are deleted.|
|fps||25||The Frames Per Second for the final rendered animation. This setting is used in both the render from Synfig, and the render from kdenlive.|
|kdenlive||storyboard.kdenlive||Name of ther kdenlive file to render. Typically this will default to storyboard.kdenlive – however it can be renamed to whatever is required.|
|fix||False||Fix is a boolean attribute. Pass in the parameter fix (which sets that to true), and the system will only fix the resource locations.|
Configuring the Snail Tales Build Job
The main configuration for Jenkins is performed through the Jenkins web-based front end. The exact configuration is based mainly on what plugins are available. For example: If you install the Chuck Norris plugin (and why wouldn’t you), then you can add a post build action, to activate Chuck Norris.
Jenkins stores the configuration for a job in the /var/lib/jenkins/job/ folder, in a config.xml file.
There were a number of plugins that were installed to make this possible :
This plugin adds Apache Ant support to Jenkins
This plugin integrates the Bazaar Version Control System with Jenkins.
ChuckNorris plugin displays a picture of Chuck Norris (instead of Jenkins the butler) and a random Chuck Norris ‘The Programmer’ fact on each build page.
This plugin allows you to store credentials in Jenkins.
Integrates Jenkins with CVS version control system using a modified version of the Netbeans cvsclient.
This plugin makes it possible to set an environment for the builds.
Adds the ability to monitor the result of externally executed jobs.
Shared library plugin for other Git related Jenkins plugins.
This plugin integrates GIT with Jenkins.
This plugin provides GitHub API for other plugins.
This plugin integrates GitHub to Jenkins.
This plugin adds Javadoc support to Jenkins.
Allows JUnit-format test results to be published.
Adds LDAP authentication to Jenkins
This plugin allows you to configure email notifications. This is a break-out of the original core based email component.
Offers matrix-based security authorization strategies (global and per-project).
Multi-configuration (matrix) project type.
Jenkins plugin for building Maven 2/3 jobs via a special project type.
Uses the OWASP Java HTML Sanitizer to allow safe-seeming HTML markup to be entered in project descriptions and the like.
Adds Unix Pluggable Authentication Module (PAM) support to Jenkins.
This plugin allows build steps to be performed before the SCM step performs an action.
This plugin provides a new enhanced API for interacting with SCM systems.
This plugin allows you to store SSH credentials in Jenkins.
This plugin allows you to manage slaves running on \*nix machines over SSH.
This plugin adds the Subversion support (via SVNKit) to Jenkins.
This plugin adds an additional dialog box in every page, which enables people to contribute localizations for the messages they are seeing in the current page.
Allows you to connect to Windows machines and start slave agents on them.
Setting up a hook into Github.
- In the Jenkins job – I clicked the option to “Build when a change is pushed to Github”
- In the Github project – I set up the Jenkins hook URL to http://addressofserver/github-webhook/
I then pushed a version of the project to GitHub, and could see that the job was queued to start processing.
Doing something with the video
The output of the script – an MP4 file – will exist on the build server until the next build overwrites it. A better solution might be to put this somewhere where this can be stored. Typically this can be done be creating a deployment step and taking the produced file and putting it somewhere. In some organizations, this could be a system such as Artifactory, and there is an Artifactory plugin for Jenkins – so this is a possibility, but with a video output, perhaps the best solution would be to upload this video to a video hosting platform – such as YouTube.
Now, there is no YouTube plugin to allow Jenkins output to be pushed directly to YouTube, but there are projects – such as youtube-upload that allows you to send a video via a python script – and I successfully used such a script to upload a video to the Jenkins Server’s own YouTube Chanel.
My next tasks will be to:
- Prove that the script does indeed replace animatic frames with actual animation
- Look at addingYouTube support to the scargo utility folder.
- Produce some video documentation that can demo this project for OggCamp
- Assemble Avengers
- Content Packaging
- Dr who
- Open Source
- Open University
- Planet Ubuntu
- Quickly Ebook Template
- s book
- Snail Tales
- This Modern Life
- February 2017
- January 2017
- December 2016
- November 2016
- October 2016
- September 2016
- August 2016
- July 2016
- June 2016
- May 2016
- April 2016
- March 2016
- February 2016
- January 2016
- December 2015
- November 2015
- October 2015
- September 2015
- August 2015
- July 2015
- June 2015
- May 2015
- April 2015
- March 2015
- February 2015
- January 2015
- December 2014
- November 2014
- October 2014
- September 2014
- August 2014
- June 2014
- May 2014
- April 2014
- March 2014
- February 2014
- January 2014
- December 2013
- November 2013
- October 2013
- September 2013
- August 2013
- July 2013
- June 2013
- May 2013
- April 2013
- March 2013
- February 2013
- January 2013
- December 2012
- November 2012
- October 2012
- September 2012
- August 2012
- July 2012
- June 2012
- May 2012
- April 2012
- March 2012
- January 2012
- December 2011
- November 2011
- October 2011
- September 2011
- August 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010