Openstack - Coding at Scale Keynote

This morning I gave a short keynote at O'Reilly's Velocity Europe entitled 'Openstack - Coding at Scale'

See the video below.

My journey so far

July marks my sixth month working on the OpenStack project for HP. Over the past few years I have had people ask me about my journey to where I am now, especially students working on the Drizzle project. So I decided to write it today.

Back in 2007 I worked as a freelancer developing PHP code, doing DBA work and administering Linux systems. One of my biggest clients was an online magazine called The First Post, I was doing so much work for them they hired me as a full time employee under the title Technical Architect. We made great strides whilst working there and I even got the site running from a MySQL Cluster installation.

Towards the end of 2007 the magazine was in financial difficulty and there was a real risk of everyone losing their jobs over Christmas. As luck would have it one of our biggest fans was a guy called Felix Dennis who owns a magazine empire called Dennis Publishing. They bought our magazine business and my responsibilities grew from maintaining one site to many. Dennis paid for me to get several certifications including MySQL DBA and developer.

One of my friends in the PHP community called Ligaya mentioned that Sun Microsystems who had recently purchased MySQL AB was hiring MySQL support engineers. This is a cross between a customer support role and developer role since we spent a lot of time debugging MySQL source code and suggesting patches to the developers. Spending lots of time in tools such as GDB was a fun challenge, especially on platforms such as AIX.

At one of the MySQL user conference I woke up very early on the first day, switched on the news to see that Oracle had agreed to buy Sun. I thought I was still dreaming but it turns out it was very real. What followed was several very difficult months for us, the regulators were trying to make sure everything was above board and every day we waited meant more job cuts and uncertainty from the public. We were censored from replying about what was happening which was very hard for a team of very vocal people. Eventually the sale was completed and things started to change.

Oracle had a very corporate attitude way to a lot of things which made things like getting paid correctly very difficult. Several of my friends had left or were leaving so I decided it was time for me to move on. A few friends had been hired by Rackspace to work on a radical fork of MySQL called Drizzle and they offered me a place there. It was a fantastic job, real open source the way it should be, the development was very rapid and well tested. Unfortunately it wasn't to last, a couple of days after we released the first GA of Drizzle we were told that Rackspace was no longer going to fund its development. This came shortly before the MySQL user conference where I was booked to give around 8 hours of talks.

I don't remember much of my talks at the conference but I'm sure they were poor. What I do remember was the stress and depression I felt at the time caused my hair to start falling out whilst I was at the conference. We had to give the impression that all was well at the conference despite knowing that it wasn't. I saw a good friend on the project go through similar things to me. It led in the end to me losing half my hair to alopecia and being on anti-depressants for the rest of the year.

Some of my friends that left Oracle had moved on to a company providing MySQL products and support called SkySQL. I joined them providing L3 support and developing new products. I was still going through my spell of depression at the time and came to the decision that I needed a complete fresh start away from MySQL and the 'community' around it.

Whilst on vacation in the USA I was called up to interview with HP's cloud department in Seattle. Working on real open source again sounded like a fantastic opportunity and I jumped at the chance. It has worked out great so far. I am getting my own team soon, am off the medication and my hair has grown back! I work on the developer continuous integration and tooling with many great people (several of whom have had similar journeys to mine).

I often get job offers to work on MySQL again but I think it very unlikely an adventure I would ever continue.

OpenStack's Jenkins Job Filler

OpenStack comprises of many projects, all of which have code reviews via. Gerrit and every code review is tested with many tests on Jenkins.  Whilst this is excellent for code quality this can get quite insane to manage.  In fact at time of writing OpenStack's Jenkins server now has a shade under 300 jobs to manage, I don't know about you but I really don't want to be spending my days managing all of that.  Often we need to make a small change to many jobs at the same time and human error occurs.  In the last few weeks we developed and unveiled the CI team's solution to this: Jenkins Job Filler (aka. Jenkins Job Builder when I forget what we called it).

The initial version of Jenkins Job Filler was written in a combination of Ruby and Puppet scripts but unfortunately it became unmanageable very quickly, it also required a Jenkins restart on every change.  So the new version has been writing in Python and YAML.  There is still some puppet behind it, but only to execute the Python script on every run.

How does it work?  There are two ways of specifying a project, one way is to have a YAML section in a file for every job, the other is to use a template.  With the template system this is all that is needed to specify all the jobs for Nova:

project:
 template: 'python_jobs'

values:
  name: 'nova'
  disabled: 'false'
  github_org: 'openstack'
  review_site: 'review.openstack.org'
  publisher_site: 'nova.openstack.org'

Much simpler than hacking XML files!

Behind this there are python modules which specify each part of a standard XML file and the template YAML file which specifies which modules should be used for each job and the parameters for each job.  It then injects the resulting XML straight into Jenkins using the Jenkins API.

This system is currently in use for all of StackForge and most of OpenStack (the last few remaining jobs will be converted over the next couple of weeks).  Now if we want to make a change to one job in all projects we can do so in one go as a tiny commit.  This is made much simpler for OpenStack due to the standard Project Testing Interface that projects should follow.  But it is flexible enough to work with non-standard jobs and projects.

To find out more see the Jenkins Jobs Documentation or the code in the OpenStack CI Puppet repository.

Juniper VPN in Fedora

Whilst I am not a fan of Juniper Network Connect in the last few years I have had to connect to several networks that use it.  The biggest problem with it is that it is a combination of Java and a 32bit C library which will not work when executed with a 64bit version of Java.
There are other ways of connecting such as the Mad Scientist script but if you use things like two-factor authentication this will not work.  So I brought together things I have learnt from web postings about getting it to work in Ubuntu and have created these steps.  They work in Fedora 17 and should work in 16 too:

Step 1

We need to install OpenJDK Java, we also need xterm for the root password during installation:

sudo yum install java-1.7.0-openjdk.i686 java-1.7.0-openjdk.x86_64 icedtea-web xterm

Step 2

Now we need to tell Java to execute the Juniper binaries in 32bit mode but everything else in 64bit mode, to do this we first rename the 64bit Java binary:

cd /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/
mv java java.orig

Then create a replacement file called “java” with the following:

#!/bin/bash
if [ $3x = "NCx" ]
then
    /usr/lib/jvm/jre-1.7.0-openjdk/bin/java "$@"
else
    /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java.orig "$@"
fi

Finally make this executable:

sudo chmod +x java

Step 3

Log on to your VPN's website as normal, an xterm session will pop up for your sudo password the first time you use this to install it.

New Gerrit and Jenkins styles are live

After a bit of tweaking and improvements following feedback from various people the new changes to Jenkins and Gerrit are now live. The changes will also be rolled out onto Stackforge in the next 24 hours.

I've had a lot of great feedback in the few hours the changes have been live, many love it and some have suggestions for improvements. This is great, we know the look could use a bit of refining here and there. What makes this truly awesome is that the styles are kept in an OpenStack project with which anyone can file bugs or send patches up to Gerrit.

To modify the styles simply grab the openstack-ci-puppet repo and look in the Gerrit and Jenkins modules for the files. If you wish to file a bug, please do so in the OpenStack CI bugs page.

Thanks again for the great feedback so far. Hopefully this has made the lives of many of you that little bit easier (or at least stop your eyes bleeding).

The real way to start hacking on OpenStack

I've seen recent attempts at blog posts to show how to get started at hacking on an OpenStack project.  Unfortunately they seem to have over-complicated the issue for new users.  As part of the Core Infrastructure team it is my job to make submitting, reviewing and merging code easier for developers.  This includes documenting that process, so here goes :)

1. You need a Launchpad account and need to be joined to the Openstack team.  You can also join the team of one of the many subprojects if you want to.  Make sure Launchpad has your SSH key, Gerrit (the code review system) uses this.
2. Sign the CLA as outlined in section 3 of the How To Contribute wiki page
3. Tell git your details:
   

   git config --global user.name "Firstname Lastname"
   git config --global user.email "your_email@youremail.com" 

4. Install git-review. This tool takes a lot of the pain out of remembering commands to push code up to Gerrit for review and to pull it back down to edit it. It is installed using:

   pip install git-review

   Several Linux distributions (notably Fedora 16 and Ubuntu 12.04) are also starting to include git-review in their repositories so it can also be installed using the standard package manager.
5. Grab a tree to hack on, for example for Nova you would do:
   

   git clone git://github.com/openstack/nova.git

6. Checkout a new branch to hack on:
   

   git checkout -b TOPIC-BRANCH

7. Start hacking
8. Run the test suite locally to make sure nothing broke
9. Commit your work using:

   git commit -a

   or you can use the following to edit a previous commit:

   git commit -a --amend

10. Push the commit up for code review using:

    git review

    That is the awesome tool we installed earlier that does a lot of hard work for you
11. Watch your email or review site, it will automatically send your code for a battery of tests on our Jenkins setup and the core team for the project will review your code. If there is any changes that should be made they will let you know.
12. When all is good the review site will automatically merge your code

Obviously nearly half of that list are tasks you will only need to perform once. So as you can see it is pretty darn simple to get started. If anyone gets stuck they are welcome to shout out in the #openstack-dev IRC channel or the OpenStack mailing list. For more information on developer work flow please see the Gerrit Workflow page, I'm not ashamed to say I still have it as a pinned tab :)

Changes coming to Gerrit's style

Gerrit is a fantastic tool for performing code reviews and automating many of the tasks which can become quite complex. Unfortunately its default style is not the most attractive thing to look at all day. Yesterday I was tasked with skinning Gerrit to make it more in-line with Openstack. This was not as easy a task as I first anticipated, mostly down to not having the ability to edit the HTML layout very much without editing Gerrit's source.

The result isn't perfect but hopefully a notable improvement which will be coming to an Openstack review site near you soon.  Here is the before and after shots: