titaniumbunker.com

Evil geniuses and world domination are 2 of our goals... we also like Dr Who

Snail Tales – Building an Animatic – Part 2

no comment

In the previous post, I had set up the Jenkins build server, along with some of the dependencies required to extract the project.  In this post I’ll look at some of the challenges that will affect the ability to build Kdenlive video using a continuous integration approach.

Ch ch ch changes….

Firstly we need to know what files have changed since the last project was built.  For example if the project was last build on Monday night, then there may have been multiple check ins since that build, meaning that there are multiple scenes that have been checked in.  Conversely it may be that there have been no changes, or a single change that involves multiple files. Bazaar will allow us to get a list of affected files for a set of revision numbers :

bzr status -r1..4

Here bazaar will list the files that have changed between version 1 and version 4.  This content can be parsed for valid files, even looking for only specific file types (for example : SIF files).

  • If the file is present then that file has either been added or modified.  This scene would need to be rebuilt.
  • If the file is not present then that file has been deleted.  This scene cannot be rebuilt, and should be removed from the Kdenlive project.

Building Synfig Scenes

Synfig offers – in addition to the Synfig-studio user front end, the Synfig command line interface.  The man page for synfig shows that a synfig file can be rendered to a sequence of images for processing into a video clip.

 

synfig -t image <<tk – add proper command line settings here>>

 

Inserting video clips into a Kdenlive project.

Kdenlive projects are xml based – which is fortunate.  It is therefore possible to perform xpath queries on the tree to retrieve all of the referenced clips and resources.  It is also possible to rewrite these references to point to new file names for the recently generated video files.  There is a bit of a gotcha here.  All paths in kdelnive are full paths, and are not relative.  This has been a problem with the kdenlive project type for a while now.  There have been bugs raised about providing the ability to store paths a relative to the kdenlive project location – however the current method of resolving the issues presented by full reference paths has been to provide a path resolution process.  This allows the user to scan the file system for matching resources, and while a useful tool for fixing these issues, it ultimately becomes less usable in an automated build environment.  Therefore we need to make sure that these paths can be calculated and re-written – and the only real piece of information we have is that the project exists at a location.  Therefore convention will be important as this will allow the script to make assumptions about where resources should be located. Kdenlive has a unique way of storing paths.  At the project level, a “root” path can be stored.   Typically this is the first 2 directory levels, but should be possible to use a single level.  the actual resources have a path – with the root removed.  This means that the combination of the project root setting and the resource location gives the full path to the resource.  This then means that we need to split a root from the resource locations that are generated by the rendering process.

 

 

 

Rendering a Kdenlive project via the command line is possible.   The man page for kdenlive

 

 

 

 

 

Animatic Baker

Dave had set up his storyboard to use still frames taken using a digital camera. These were added to the Kdenlive work space and added to the timeline in order. By default Kdenlive creates a 5 second allocation for each shot when it is added to the timeline, but these can be stretched or squashed as appropriate. While this process works well for Dave, it causes a problem for Continuous integration. The files referenced are graphic files, and replacing them with video files causes a problem for kdenlive. The solution to this is to effectively ‘bake’ the animatic, replacing each graphic with a video clip of the correct duration. A replacement video can be rendered, overwriting the storyboard video, and when the project is rendered, that animatic now contains a replaced scene.
I worked on a python script that would investigate the Kdenlive project file, and extract filenames from paths, modify paths and update those file locations to the new location. Under the covers, the Kdenlive project is an xml file, and using lxml I could extract the sections of the file I was interested, apply changes to those sections and write the file back. Loading that Kdenlive file back into Kdenlive revealed the editor had an issue loading video files when the section of the file (2 sections called producer and kdenliveproducers) was originally a png (or indeed any other graphic) file.
I am currently working on a solution that constructs and replaced image based producer/kdenliveproducer sections with new sections based on the replacement clips – and will write more updates as I resolve these issues.
The challenges of animatic baking should only be experienced once during the project, so I decided to manufacture a storyboard by hand with replaced video clips. To create the replacement videos I used the following command line:

 avconv -loop 1 -i sb1_0001.png -vcodec libtheora -tune stillimage -t 41 OUTPUT_VIDEO.ogv
Here the video codec in use is Theora, tying into the original Snail Tales aim that the entire project be completed using open source and libre standards. Naturally other codecs are available, and could be substitued. Indeed for a production build a non-lossy format should be used – and in the end I opted to use a x264 render codec, as Kdenlive was experiencing difficulties reading the OGV file produced. It is also possible to use online resource to convert your video clips :
Video conversion functionality provided by video.online-convert.com/convert-to-ogg.

Video conversion functionality provided by video.online-convert.com/convert-to-ogg.

 

Wait – this is all getting somewhat complicated isn’t it?

Well – yes. It seems that the idea of continuously building the animatic based on updates to the scene source is not without it’s own set of challenges. – So how about a quick recap of where we are at.

  • I created a folder structure that will store the scenes, the artwork, the assets – basically everything needed to work on or build Snail Tales. The reason for that was that this allows animators and build services to consume the same content, and working with a disconnected system such as BZR would allow animators to add to their local repositories before pushing their scenes to the server.
  • I also created a copy of the animatic Kdenlive project, replacing the image shots with video shots. This would allow an animator to replace the finished scene into the animatic.
  • I’ve started writing a script that would automate much of this process (a process I call Baking) – but that’s still in progress.

So what else do we need to complete this utopian vision where animators check changes into a repository and a new animatic is produced automagically?

Scripting a build

the animatic needs to be build without user interaction. Ideally we should be able to run a script and the result should be a a video file is produced. Kdenlive is built on the MLT framework – a set of command line tools for interacting with video files, and kdenlive happens to expose that command line goodness, and even allows us to script a render through the rendering option.

KdenLive - Generate render Script

Kdenlive rendering options can generate a script for your render

Using this option I created a script and resource file – the imaginatively named animatic.sh and animatic.mlt.

 

Rewriting kdenlive Paths

If only kdenlive supported relative paths then that would be pretty much all we’d need, but unfortunately it doesn’t. What this means is that when Jenkins attempts to run the build script it is going to attempt to load video files from whatever location is in the file – in my case /home/mike/projects/snail-tales/animatic – which is fine if the server happens to be on the same server as the source matterials (which in my case it is). What we need to do is to modify those addresses to point to the workspace where jenkins checks out its files to build. Luckily we can do this in the build script within the Jenkins job for now, but I will probably move that functionality to a Python based solution, as much of the functionality relating to fixing paths would already be required for the animatic baker.
Accepting the current limitations on the solution, let’s get cracking with installing Jenkins then.

Installing Jenkins

Installing Jenkins under ubuntu is incredibly easy. I found a page by googling “install jenkins onto ubuntu”.
ultimately it boils down to :

wget -q -O - https://jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -	 	 
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'	 	 
sudo apt-get update	 	 
sudo apt-get install jenkins

Once installed you can access jenkins by pointing your browser to http://localhost:8080.
Screenshot from 2014-08-30 01:22:05

I installed the bzr source control plugin Manage Jenkins > Manage Plugins > Available > Bazaar Plugin.
Next I created a job.
SnailTales build job

This is where the process gets more complicated. As I mentioned before, In order for Jenkins to build out project correctly it will need to replace certain paths within the Kdenlive generated files – mainly relating to where the scene clips can be located, and where the output files should be saved. Therefore I wrote the following Script

#!/bin/bash 	 	 
sed -i "s;/home/mike/projects/snail-tales/animatic;$WORKSPACE/animatic;" $WORKSPACE/animatic/animatic.kdenlive	 	 
sed -i "s;/home/mike/projects/snail-tales/animatic;$WORKSPACE/animatic;" $WORKSPACE/animatic/animatic.sh	 	 
$WORKSPACE/animatic/animatic.sh

This script uses sed to replace all references to the original location, to the new location based on the Jenkins Workspace. Jenkins has a concept called a workspace, which is an area on disk where jenkins will perform it’s work – including checking out bzr source code. Therefore the addresses for assets can be changed to point the folder location within the workspace rather than the original file location.
I initially create a schedule to build every 5 minutes – however I didn’t take into consideration that this would be accomplished by polling the source control system. This created a situation where the output was being re-rendered every 5 minutes whether it needed to or not. It also had an issue that if the build took longer than the job interval, that there would still be a second instance of the build started. This could have theoretically been resolved by forcing the job to use a specific executor – however I plan to replace this with a bzr hook. The act of checking in a change (to a particular file type) will force the rebuild of the animatic, or would enable the overnight scheduled build of the animatic – depending on how the system is configured. Trying to create a cron style schedule string is not exactly fun, and therefore I used a utility page at http://cron.nmonitoring.com/cron-generator.html to create the crontab style string for scheduling the build.

Gotchas

Kdenlive stores all path information as absolute paths – which when you are running the Jenkins server on the same machine mean that paths are still satisfied. But put that bzr repository on a network share, or even into lauchpad and those asset locations can’t be guaranteed to work. This is why the job re-writes part of the kdenlive files. As these files are xml files, it might be better / more efficient to redevelop these replacement scripts with something that uses xslt to update paths based on a selection criteria. Currently the mlt file still contains references to the source folder – these need to be modified in a similar way to the existing script – however there are slight changes. The MLT file has root attribute along with property elements with resource attributes. Together these make up a whole path. Having worked with xml for the Ubuntu Quickly Ebook Template, then there might be a possibility that some of the code could be re-used to modify the mlt files more effectively.

While developing the Snail Tales project structure, the test project experienced an issue where jenkins appeared to cut off early. Generated files were 14.3mb rather than 14.9mb. I believe this is a disk caching issue, or possibly related to Java.
For those that are interested – here’s what the animatic looked like once built :

Creta marine : Internet Access Kiosk

no comment

So – I bit the bullet and decided that I would invest some money in some internet access.  I stumped up the 2 Euros to access the machine – and I started thinking about how you make such a machine secure and usable.


Read more..

Error message in the wild

no comment

Saw this 404 error at Birmingham airport…

image

image

Grooveshark – Gone!

no comment

By Hermanus Backpackers (Great White Shark Cage Diving) [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons

So now, grooveshark  has gone. The music Industry would no doubt want me to go back to spotify – I’ll check it out, but if the user experience was as bad as the last time I used it, then I will probably not bother listening to streaming music over the internet again :(

Project:Snail Tales update

no comment

For those of you in the know, for the past couple of years I have been working on an animated short film. It’s a long process to make a short animation and with lots of assets to keep track of. I use a production chart to keep tabs on everything. here is a snapshot of the production chart as it stands:

Now the eagle-eyed amongst you will notice that it’s a spreadsheet In the past I have tutted and rolled my eyes when people have complained that when they use a spreadsheet to catalog their DVD collection, they couldn’t per pixel scroll, it would snap to the nearest cell.  And then a patiently explain that a spreadsheet is not designed to catalog a collection of DVD’s, A spreadsheet is really good a totaling columns of numbers and/or applying formulae on them. A DVD catalog is best done with a database.

Yes I know I should use a Database to store the production chart. It is a more effective way to store this information. Each scene is a record that can have a series of fields applied to them. we could poll the database for complete scenes and get an accurate percentage of how much of the film is animated or rendered  or needs work(etc)

thing is I am, to my own surprise, a little bit old school. I learnt to breakdown sound using a mixing desk, and large sheets of paper, jogging through soundtracks, listening for the pops and whistles and decoding them into the phonemes that made up the characters speech. and this is a digital equivalent of the old school way of creating a production chart – It’s a digital analogue of an Analogue er.. analogue

today , kids examine wave forms or use software tools to provide easier breakdowns, and whilst I like them and do use them a fair bit, sometimes , I think that younger animators, fresh into the field, are lacking some of these old school skills.

part of my old school curmudgeon-ness is the creation of dope sheets and production charts. there was something exciting about transferring your sound breakdown to a dopesheet ready to animate, it was a prelude to the storm of creation that leads to the initial pencil tests. I loved the way the Production charts would fill up with checks and notes becoming more full as the deadline approached.

working on this project has been great fun. the biggest problem has been scheduling the time to make the animation and learning to use the software. Part of that has been learning some of the limitations of the software and the creation of new software tools to allow me to work with the software the way I want to work with it. I was using synfig stage last night and it struck me I have talked a lot about it at Oggcamp and other tech shows without really showing it. I started using it and it worked straight away (more or less) and so showing it working didn’t feel important because it actually was working. I suppose I should make a video demonstrating the tool and the problem it solves.

Priority though is on the film. Right now with about 16  scenes left to animate there’s a definite feeling its starting to come together as a film and part of me will be glad to get it finished, to move on to the next thing. part of me also misses my old school beginnings.  and I hope maybe one day in the future, I will do a proper old school 2D short using an actual pencil on real paper.

 

We have a reader!

no comment

That’s right – you read that right… We

HAVE A READER!

I received an email from a reader –

Hi Mike,

I got an issue with my bought on O’Reilly books’ reading in Amazon Cloud Reader.
When I was searching Google for solution I found your post on “http://titaniumbunker.com/?p=2707″.

I read your discussion with Amazon support team and carefully tried to apply their suggestion. As result I could load “*.azw” books to the app “Amazon reader for PC” that were bought outside the Kindle store as a MOBI format.

Sorry for this email. But if this info is still interesting for you, I made following steps to have a possibility to read the books:
1. I bought the books in MOBI format named “ng-book-25.mobi”.
2. I sent the book in MOBI format to my Kindle email. As result the book named “ng-book-25.azw” appeared in my Amazone Cloud Drive.
3. I Downloaded “ng-book-25.azw” from Amazone Cloud Drive to hard disk of computer.
4. I installed “Amazon reader for PC” and set the folder for local content in its menu “Tools | Options | Content” to my special local folder “D:\Documents\My Kindle Content”
5. I put downloaded “ng-book-25.azw” book to this folder and reopened the app.
6. The book appeared in the “Downloaded items…” root of the reader and available for reading.
7. I think you may put any other supported books’ formats directly into this folder as well.

Regarding Amazon Cloud Reader, it is possible to hold there only the books bought in Kindle Store. In this case such a book appears in Amazon Cloud Reader automatically.

Best regards,
Denis Levushkin

Wow – A real live human has read our (or part of) our blog.

I responded to Denis – and his comments are correct – you can upload a book from O’Reilly and get it into your kindle library, and into Amazon Reader for PC, and I have managed to get that to work.  So his instructions are 100% correct.  As a bonus, you can link your O’Reilly and kindle accounts, and get O’Reilly to send newly bought books directly into your Kindle Library.

 

Message from the Brig

no comment

Well I’m writing this from the Titanium Bunker brig.  Conditions here are terrible wonderful, and I couldn’t be happier.  There’s practically no signal an opportunity for me to relax in wonderful surroundings and they only have strawberry Ice cream, and that’s like my 3rd favourite ice cream have a wonderful menu for me to try.  The security bods geniuses weren’t too happy that I was able to defeat the access control systems using a paperclip and some duct tape have been wonderful in working with me to see the error of my choices, and to further improve the access security controls.

Anyway I’ll write some more if I get chance, but I worry that my communications will be censored by the security team and I have no concerns that my communications are in anyway monitored of censored.

 

 

Busted – Update

no comment
Titanium Bunker Incident Report Form

Oh I’ve got you Mike… Bang to rights!

I filled in the paperwork today to record the recent security ‘incident’.  I’m not a vindictive man however, and I therefore I don’t intend to apply for the death penalty. 😉

BUSTED!

no comment

so I was reading about last weekends sci fi weekender over  on the daily post  panel1   about halfway down the article there was a slideshow about the weekend. so I clicked it. After all I love a Cosplayer

panel2looking at image 3 there struck me as something not quite right….

panel3…Lets zoom in a bit shall we?

panel4OMG! it’s Mike, Rachel and the future Mr & Mrs Byrne! and mike said he was at a .net conference!

It’s obvious mike went AWOL from the bunker. Don’t worry TitaniumBunker security has been informed.

 

whispertoacop

 

Bald Sherlock

no comment

I was going through some old files when I found this.  pretty sure there was a reason behind it-just cant remember why

baldsherlock