titaniumbunker.com

I popped onto the Lloyds Bank site to download a form relating to a future change of address and I spotted this issue…. Can you see it?…  So I reckon some people are probably having a bit of a knowing chuckle while the rest are probably scratching their heads and wondering what all the fuss was about.

I’ve highlighted the error in yellow – and here’s a close up

 Ok – so what’s going on here?

Which looks surprisingly similar to the content extract Lloyds presented on their search screen.  So it seems that the PDF file content is indexed using a PDF viewer, and rather than indexing the real content of these documents, it has used this incompatibility message as the description for the PDF content.  We can further confirm this by searching for words from this document that are unlikely to appear in typical banking systems – so searching for the term “Linux” gives 12 results, all of which feature the first part of the Linux sentence.

 

Searching for “Please Wait” gives 126 documents that are equally badly indexed – some don’t even have a document title.

As to why some have titles and some don’t I would have to guess that documents are uploaded through some form of content management system – and that these document may originally been in an older version of PDF (say – 1.4). These documents have since been updated – possibly with new interactive form filling features, and been re-indexed. The indexing system uses a version of PDF viewing which doesn’t support version 1.7 – and therefore indexes the incompatibility message as if it were document content (after all the viewer wouldn’t know whether it is showing content, or a message).

Other documents may have been created in a newer version of PDF, and the title attributes may not have been accessible when the document was indexed.

Update – I’ve tweeted this bug information to Lloyds.

.@AskLloydsBank Hi – I think I've found a bug in your website. I've written a blog post about it – https://t.co/9TZjupJc74 #PDF_IndexingBug

— Mike Hingley (@computa_mike) May 26, 2017

Well, it seems that yet again the Oscars are incapable of giving away 24 gold-plated statues to a load of rich actors in designer gear without some kind of controversy. This year Bonnie and Clyde – Faye Dunnaway and Warren Beaty managed to inadvertently inject anarchy into the awards do by opening the wrong envelope. resulting in La La Land winning best picture when it should have been Moonlight. It transpired, Beaty had been handed the envelope for the previous award for Best Actress, won by Emma Stone for La La Land. The Academy have announced an inquiry but here are the basic facts

apparently, there are 2 people backstage responsible for handing the envelopes from one of 2 sets of envelopes to the presenters. You have to assume the envelopes were marked with the award on there and yet 2 people failed to give an envelope to someone who didn’t even bother to check he had the right envelope. It seems that the problem can be summed up as follows. unused envelopes had contaminated the award pool. either the old envelope opened by Leonardo Di Caprio had somehow made its way to Beaty, or the alternative Best actress awards had been handed out in error. Now I didn’t watch the awards but I would assume there would be some stage directions managing where presenters and winners would enter and exit the stage. Here’s a brief intro to stage direction terms

 

in this diagram, we can see that stage left (SL) and stage right(SR) is relative to an actor standing on stage and facing the audience that forestage (FS) is closer to the audience and backstage(BS) is further away. Now a lot of these awards will also have an elevated staircase that the presenters enter sometimes from SL some time from SR but always from an elevated position, meeting in the center and walking down a shared staircase to centre forestage. Once at the podium the presenters ad-lib and read the autocue to announce the nominees for the award, the envelope is opened, the winner is read, the winner receives the awards, talks for 30 seconds then returns to their seat, the presenters then also leave. Because the next presenters should already be in position, ready to come on it is doubtful the presenters will exit the stage the same way they came in. so here are our stage direction options:

• presenters walk in either from stage right
• presenters walk in either from stage left
• presenters walk in either from a central staircase

once they have given out the awards they exit

• If presenters entered Stage Right, then exit Stage left
• If presenters entered Stage Left then exit Stage Right
• If presenters entered from central staircase, then exit either Stage left or stage right

If there are 2 people handing out the envelopes, then I suggest that 1 of them hand out the envelope at the start, once the envelope has been handed to the presenter, the matching envelope from the alternate set is destroyed. The other person waits for the actors to exit, takes the envelope and card from the, rips them in half and places them in a bin before ticking them off a list.there should be no contamination of envelopes and if the envelopes are in order then the person handing out the envelopes merely has to hand the next available envelope to the presenters, something a robot could do.

in future, the best way to do this possibly would be to announce all the nominees on the autocue and have just a blank piece of paper in the envelope. then there’s only one person responsible for messing up, namely the person who enters the text for the autocue. The text could be signed off by the awards committee either beforehand or in situ ( to be honest, I can’t believe there is no such thing as an autocue that reads from an external file )

This kind of falls into our examination of award ceremony structure. We did look into how an open source award ceremony could be made. our method removes the need for envelopes, mix-ups and incorrect nominees as the award ceremony is generated from the votes dynamically, the only thing you need is someone to press the buttons to switch slides and then bring up the winner at the end. its possible that the next slide preview could be piped to an autocue system or the text from each slide scraped into the autocue giving the announcer the text they merely have to read.

Who needs to log stuff anyway…I wonder what that other process is?

Hi Everyone,  I know its been a while since I last posted, but there are reasons – I will post more – promise

In the meantime, I was going through some of the draft blog posts I had started but not finished. and came across this one, So I dusted it off and finished it off.  I don’t know if the proposed legislation has been implemented or if it’s been altered since I started it.  In any case here it is:

 

 

On 16th March 2016 , the Chancellor of the exchequer, George Osbourne unveiled a new budget,chief among the new policy was a levy on sugary soft drinks – The nation is grip of an obesity epidemic and fizzy, sugary drinks have been identified as the culprit. according to other press agencies, Jamie Oliver did a little victory dance outside the Houses of Parliament as it was announced.

 

Read more..

In the last post about making the magic work I discussed how to get continuous integration working with the selfietorium project, and getting it building and deploying a deb file.

This time I’m talking about measuring code quality through SonarQube and Codacy.

At (real) work, we’re planning on using SonarQube to measure code statistics – its a tool that will tell you whether your variable names match coding standards, or whether your code is duplicated, or has unused references etc.  But I found out that SonarSource host an instance of SonarQube that can be used to analyse open source projects.  As selfietorium is an open source project, I signed up.

For those looking to a more private solution, then it is possible to run your own sonarqube server, and that might be a topic for a future post – but for now I’m going to set up SonarQube.com to analyse selfietorium.

Logging on to SonarQube :

SonarQube uses github authentication, so connecting is easy. Once you are logged in you’ll need to create a security token.  SonarQube works by having code pushed to it – so in the previous blogpost we used Travis CI to push the code to SonarQube.  To make this happen we need a security token.

To create a security token :

Once you have authenticated with SonarQube and logged on, click your name in the top right of the page, and select “My Account”.

• From here click “Security”
• Click Generate to generate a token – give your token a suitable name.  Once your token is generated make a note of it – you are going to need it later.

Back in Travis CI, the sonar-scanner line instructs a plug in to push the code to sonarqube using the configuration section and a new file that needs to be added to the project root:

sonar-project.properties

Selfietorium’s configuration file can be found here.  SonarQube does have documentation about setting up the properties file, and it isn’t hard to set up.

We can now use the token we got from SonarQube to tell Travis-CI what to authenticate itself as.  This is stored in the environment variables section, using the techniques I touched on in previous post, in particular the section on “Keeping Secrets”

Next time you build your project, it will be pushed to SonaqQube (along with the sonar-project.properties), and analysis performed against the code.

SonarQube is a great tool, but it doesn’t give us what we really want – a nice graphic we can add to our project read me – after all, that’s what’s important right?

Using Codacy

Like SonarQube, Codacy uses github for authentication.  To set up a project for analysis, it’ just a case of clicking your project and clicking go.  It’s a much simpler setup that SonarQube.  Getting that all important badge is also a breeze.  Click on the Settings button from the dashboard.
From here you can generate markup for different documentation systems including html, rst, and markdown.  Just copy the Markdown and paste it into the appropriate document on your github repository and now you’ll get a badge rewarding you for making the code better.

 

It has been bought to my attention that the initial version of this post incorrectly identified the titanium bunker department responsible for the development of the version 1 selfietorium enclosure as titanium shed. This was obviously incorrect, as titanium shed was the original project code for the department now known as titanium workshop. Accordingly the credit for this work should be titanium workshop.

Titaniumshed has now produced an initial version of the selfietorium enclosure.

I’ve been focusing my efforts on the selfietorium lately and in particular how to combine all the various support systems with GitHub, where the source is stored. This blog post details making the magic work: to get Continuous Integration, build, package and release uploads working.

Continuous Integration is the foundation from which the other support services will hang.  There’s no point in performing code analysis on code that doesn’t build or pass its tests.  So, let’s get started.

In previous projects – like Snail Tales – we have created Jenkins installs and created build scripts for all of this to work.  For the selfietorium project we are using Travis-CI.

Selfietorium is a raspberry pi based Python project, and there is great support in Travis-CI for Python – some languages such as c# are not 100% supported, so Travis-CI may not be suitable for all uses.  Before you start looking at using Travis-CI for your solution, you should probably check that the language is supported by taking a look at the getting started page in the Travis-CI Docs.

Techies amongst you might be thinking

Mike – what are you going to build?  Python is an interpreted language – there is no compiler for Python

And that’s true enough.  I aim to use the Travis-CI build system to run my unit tests (when I write some) and package my Python code into a Debian .deb file to allow easy installation onto a Raspberry Pi.

So let’s get cracking

To start with, you’ll need an account on Travis-CI.  Travis-CI uses GitHub for authentication, so that’s not too difficult to set up – just sign in with your GitHub account.

Now that you have an account what do you do next?  There are a couple of things you need to do to make your project build:  Create your project within Travis-CI and create a .travis.yml file

The .travis.yml file contains all of the steps to build and process your project, and it can be somewhat complicated.  What is amazingly simple though is setting up a GitHub repository to build.  Travis-CI presents me with all of the repositories that are capable of being built.  From here I picked the TitaniumBunker/Selfietorium repository, and that was pretty much it.

Once your repository is set up it needs to be configured – the docs are an absolute must here. There is no IDE to manage your configuration – all that stands between build success and multiple frustrating build failures is you and your ability to write a decent .travis.yml file.

Nothing will build until you next push something to your GitHub repository.  Push something to your repository and Travis-CI will spring into life, and potentially fail with an error, probably looking something like this:

Worker information
hostname: ip-10-12-2-57:94955ffd-d111-46f9-ae1e-934bb94a5b20
version: v2.5.0-8-g19ea9c2 https://github.com/travis-ci/worker/tree/19ea9c20425c78100500c7cc935892b47024922c
instance: ad8e75d:travis:ruby
startup: 653.84368ms
Could not find .travis.yml, using standard configuration.
Build system information
Build language: ruby
Build group: stable
Build dist: precise
Build id: 185930222
Job id: 185930223
travis-build version: 7cac7d393
Build image provisioning date and time
Thu Feb  5 15:09:33 UTC 2015
Operating System Details
Distributor ID:	Ubuntu
Description:	Ubuntu 12.04.5 LTS
Release:	12.04
Codename:	precise
...
...
...

There’s a lot of cruft in there – but the lines that are interesting are:

• version – The version line hints that the Travis-CI worker code is on GitHub.  It is.
• Could not find .travis.yml, using standard configuration. – The build fails to find a .travis.yml file and defaults to building a Ruby project.
• Description: Ubuntu 12.04.5 LTS – the build workers seem to be Ubuntu based.
• Cookbooks Version a68419e – Travis cookbooks are used with chef to set up workers

.travis.yml file

The .travis.yml file is effectively a script that executes as the build life cycle executes. The Customizing the Build page says that a build is made up of 2 main steps

1. install: install any dependencies required
2. script: run the build script

These 2 sections are essential for any .travis.yml file. There can be more than just these 2 sections, and the Customizing the Build page details a whole bunch of extra steps that can be added to your .travis.yml file.

The. travis.yml file for selfietorium looks like this:

dist: trusty
sudo: required

addons:
  sonarqube:
    token:
      secure: '$SONARQUBE_API_KEY'


language: python

python:
 - "2.7"
# uncomment and edit the following line if your project needs to run something other than `rake`:

before_install:
  - sudo apt-get -qq update
  - sudo apt-get install -y build-essential devscripts ubuntu-dev-tools debhelper dh-make diffutils patch cdbs
  - sudo apt-get install -y dh-python python-all python-setuptools python3-all python3-setuptools
  - sudo apt-get install -y python-cairo python-lxml python-rsvg python-twitter
  
install: true
 
script:
   - sonar-scanner
   - sudo dpkg-buildpackage -us -uc 

before_deploy:
  cp ../python-selfietorium_1_all.deb python-selfietorium_1_all.deb
tyle="white-space: pre-wrap; word-wrap: break-word;"
deploy:
  provider: releases
  api_key: '$GITHUB_API_KEY'
  file: 'python-selfietorium_1_all.deb'
  skip_cleanup: true
  on:
    branch: master
    tags: true

So let’s break down what this script does.

dist: trusty
sudo: required

addons:
  sonarqube:
    token:
      secure: '$SONARQUBE_API_KEY'


language: python

python:
 - "2.7"

This section is the pre-requisites section. It tells Travis-CI that the worker that is going to run this script should be an Ubuntu 14.04 LTS (Trusty Tahr) based machine.  Travis-CI will build on either a Virtual Machine environment (with sudo enabled), or a Container – which is I believe based on Docker.  The issue with docker is that while it takes seconds to provision a container based environment, it currently doesn’t have sudo available to it, meaning that performing activities using sudo (for example, installing build dependencies) is not possible in a container based environment.  The Travis blog does state that:

If you require sudo, for instance to install Ubuntu packages, a workaround is to use precompiled binaries, uploading them to S3 and downloading them as part of your build, installing them into a non-root directory.

Now I still have some work to do around dependency resolution – I think it is possible to trim the number of dependencies right down. At the moment the build system installs all of the runtime dependencies which might potentially be overkill for the packaging – however they still might be needed for unit testing. Further work is required to look into that. If these dependencies can be removed, then the build could potentially be done in a container, speeding up the whole process. I can almost hear the other fellow techies…

But Mike, why don’t you use Python distutils, and use pypi to install your dependencies?

A fair question.  Using pypi would mean that I could potentially install the dependencies without needing sudo access – the issue is that python-rsvg doesn’t seem to be available on pypi, and only seems to be available as a Linux package.

In this section I’m also telling Travis-CI that I would like to use SonarQube to perform analysis on the solution ^[1] and that the solution language is Python 2.7.  I think the general opinion of developers out there is:

Use Python 3 – because Python 2 is the old way, and Python 3 is the newest

I’d like to use the new and shiny Python 3, but I fear that there may be libraries that I am using that have no Python 3 implementation – and that fear has led me back into the warm embrace that is Python 2.7.  I plan to perform an audit and determine whether the project can be ported to Python 3.

before_install:
  - sudo apt-get -qq update
  - sudo apt-get install -y build-essential devscripts ubuntu-dev-tools debhelper dh-make diffutils patch cdbs
  - sudo apt-get install -y dh-python python-all python-setuptools python3-all python3-setuptools
  - sudo apt-get install -y python-cairo python-lxml python-rsvg python-twitter

In this section I am installing the various Linux packages required to perform the build.  These are standard commands for installing packages onto an environment.

install: true

And here the installation does nothing.  As I said at the top of this article, there is no build for Python programs.

script:
   - sonar-scanner
   - sudo dpkg-buildpackage -us -uc

Right about here is where I would be running any unit tests – but I don’t have any yet.  This script then sends the code to SonarQube – a topic for a future post – and then calls dpkg-buildpackage to create the binary package.  At the end of this step we have a deb file that could potentially be deployed.

before_deploy:
  cp ../python-selfietorium_1_all.deb python-selfietorium_1_all.deb

Before I deploy the deb file, I need to copy it, so I copy the generated deb file into the current working directory.

deploy:
  provider: releases
  api_key: '$GITHUB_API_KEY'
  file: 'python-selfietorium_1_all.deb'
  skip_cleanup: true
  on:
    branch: master

    tags: true

Finally, we deploy the file.  The provider: releases line tells Travis-CI to use the GitHub Releases provider to push the build artifact to a GitHub release.

It uses a secret API key to gain access to the project releases.  The file is the name of the generated file, and the skip_cleanup prevents Travis-CI from resetting the working directory and deleting all changes made during the build.  The on section controls when files are deployed.  With this setting, only changes made to the master branch, that are tags trigger the deployment.  GitHub releases are actually tags, so creating a release creates a tag on a branch.  For selfietorium we create releases on the master branch.  The release deployment then pushes the build artifact to GitHub effectively offering it as the binary file for that release tag – and uploading it to the GitHub Release.

Keeping Secrets.

In order for Travis-CI to upload your build artifact, it will need to identify itself, and to do that we create a Personal Access Token.  Using this Token, the GitHub Releases provider can now communicate with GitHub as if it was you.  We can’t just add the GitHub token to our .travis.yml file.  Well, I suppose we could, but then we shouldn’t be surprised if other files start appearing in our releases.  The .travis.yml file is publicly available on GitHub – so we need a way of storing the token securely, and injecting it into the script during execution.  Travis-CI offers an ability to store environment variables within a project.  These variables are hidden when the log files are produced if you clear the display value in build log check box.  To use that variable in your .travis.yml file you’d refer to it like this:

$GITHUB_API_KEY

Grabbing a Status badge

Within the Travis-CI project settings screen, clicking the status badge offers the ability to generate suitable markdown for GitHub.

Quick Recap.

So what we’ve done so far is:

• Configured Travis-CI to build when we push to the repository.
  • Eventually this will allow for unit tests to be run – but at the moment there are no unit tests for selfietorium.
• Configured Travis-CI to package the application into a .deb file when a release is created.
  • Releases are effectively tags within a git repository.
• Configured Travis-CI to deploy out build artifact back to GitHub using a Personal Access Token.
  • Personal Access Token is securely stored in a Travis-CI environment variable.
• We’ve created some spiffy markdown for a status badge that we can incorporate into our repository markdown.

In a forthcoming as yet un-written post, I’ll document how to set up the packaging folder so that selfietorium is installable and executable on a Linux system.  It will probably borrow quite heavily from this page I wrote about content packaging.

Useful tools

Travis WebLint – check your Travis configuration for obvious errors.

Footnotes

• ^[1] I’ll return to that in a later post when I’ll talk about Continuous Inspection

I know it’s sad times for staples UK – I spent many a happy time in staples, refreshing my manilla folders for my family research – but I can’t help it’s a little early for it all to start to fall apart.

My office chair – actually bought from Staples only a few years ago is starting to look its age, and I thought about replacing it – so I clicked on the “See all Deals” button under “Big Chair Event” and presented with a list of manager and executive chairs.

 

Now I’m not really a manager type – I like to get my hands dirty  (in as much as I don’t like to get my hands dirty – that’s why I work with code) so I was thinking about a mesh chair.  So I clicked on Mesh Seating :

Also missing are Draughtman Chairs. Interestingly I can find a mesh seating section  – http://www.staples.co.uk/mesh-seating/cbk/670.html

So what’s happening?

Well – comparing the draughtsman, mesh seating and ergonomic chairs links – against the working links, it seems that the culprit seems to be : cm_sp.

For example – here is the failing Mesh Seating link :
http://www.staples.co.uk/mesh-seating/cbk/670.html&cm_sp=W16_11_017_02UK-_-Na-_-Na?web_track_id=135829767&position_id=2&promo_code=989989999&lcb=10

And a slightly modified (and now working)  mesh seating link :

http://www.staples.co.uk/mesh-seating/cbk/670.html?cm_sp=W16_11_017_02UK-_-u_ad_4_href&web_track_id=135829704&position_id=2&promo_code=989989999&lcb=10

The highlighted Na-_-Na looks suspiciously like Not Applicable, or potentially “NaN” truncated to fit.