Monday, 24 December 2012

A MySQL Christmas present - Libdrizzle 5.1.0

Brian Aker and I have been working hard in the last few weeks to give you a great Christmas present, Libdrizzle 5.1.0.  The MySQL compatible, BSD licensed C connector (so static compiling with commercial software gets the thumbs up!).

The latest changes include:
  • A server-side prepared statement API
  • Improved binary log API
  • An example binary log remote retrieval utility using the binlog API called "drizzle_binlogs"
  • A new build system, DDM4 which is used by Gearman and Memcached
  • Many bugs fixes
The source and manuals can be found on the Launchpad downloads page.  Please enjoy, feel free to file bugs, questions and hack on code on our Launchpad page.  Happy holidays to all!

Sunday, 9 December 2012

Libdrizzle Redux 5.0-alpha1 Released!

Over the past few months I have been spending my spare time on a new project.  A new version of libdrizzle which is much simpler to use and with many new features.  Today the first version of this MySQL compatible client is released, called Libdrizzle Redux.

Why 5.0?  Because Libdrizzle 1.0 and 2.0 have already been released, in packaging versions 3.0 and 4.0 used as API revisions.  So 5.0 is the fresh start.

Main Features

These are the main features of the library:

  • A BSD licensed MySQL compatible C connector, so you can statically link it with commercial software
  • A simplified API compared to Libdrizzle.  No more confusion over whether the client or library should be allocating/freeing.  There isn't a big difference to the MySQL C API for most things.
  • New documentation, PDF for now web based coming soon.
  • A new binlog reading API.  This can connect as a MySQL slave or mysqlbinlog client and get each event from the binary log.

Differences from Libdrizzle

Whilst based on Libdrizzle, Libdrizzle Redux has had an overhaul:

  • The API is a little different from the older Libdrizzle library, for the most part it has been simplified
  • The server-side API has been stripped out, this is now a client-only connector
  • The new Binlog API
  • Improved documentation

Still to Come

These is still a lot of work to do.  There are several features I want to get in over the next 6 months and improve the features we already have.  Some example of the improvements needed are:

  • Improvements to documentation with better examples and a breakdown of the explanation of the examples and how they work
  • More tests!  Testing against more platforms too
  • A different build system.  I used CMake for now for RAD purposes, Brian Aker is already working on switching this out for an autotools based system.
  • Server-side prepared statement support
  • Several new client projects using this library

This is a fully open source project, so anyone can come along and hack on it.


You can download it now from Launchpad along with the documentation.  The download page is here.

Tuesday, 4 December 2012

Slides for Connectors Talk

I could not find a way to upload my slides for today's talk entitled "MySQL Compatible Open Source Connectors" on the Percona Live website so the PDF can be viewed on Slideshare.  Enjoy!

Thursday, 15 November 2012

Upcoming MySQL Conferences

I may not work on many MySQL related things any more but there are some MySQL ecosystem events that I am participating in.

First there is Percona Live London.  This event is on the 3rd and 4th of December at The Millennium Gloucester Conference Centre.  At this event I will be giving a talk on MySQL Compatible Open Source Connectors.  This will cover alternatives to the standard connectors and how to use them, it will be useful if you find the licensing behind the standard connectors prohibiting.  There is also a 40% off discount code you can use when registering, simply enter the promo code "Come2mytalk".

On the 2nd and 3rd of February is FOSDEM.  At this event there is a "MySQL and Friends" dev room on the Sunday.  I am on the committee overseeing the talks submitted to this room and am looking forward to some great talk submissions.  One of the great things about FOSDEM is it is completely free and no registration is required.

I'm looking forward to seeing everyone related to the MySQL ecosystem at these events!

Thursday, 11 October 2012

A few days with Gnome 3.6

A few days ago I decided to upgrade one of my laptops to the current alpha (soon to be beta) of Fedora 18.  There have been a lot of changes to Gnome with the 3.6 release and there have been may critics of these changes.  I actually want to stand-up and say that I am a happy Gnome 3 user, I've used it since way before 3.0 was released and am impressed with some of the changes in 3.6.

Most of the things I like in Gnome 3.6 are subtle changes.  For example I am British but all my laptops come from the US with US keyboard layouts.  Whilst docked I connect a UK keyboard to them.  Gnome 3 supports multiple keyboards but I usually find that after I unlock the screen it has reverted back to the default layout.  Now not only does this not happen but the new settings for it make it easy to set key bindings to switch layouts.

I don't yet know whether this is a Gnome thing or a Fedora thing, but I spend most of my time connected to a work OpenVPN server.  The NetworkManager icon now shows 2 icons, one with WiFi (showing signal) and one showing that I'm connected to a VPN.
These also show on the lock screen.  Speaking of the lock screen, there is now a "swipe-to-unlock" type screen before entering your password.  If you are a heavy keyboard user like me this can easily be cleared by a press of 'enter' or 'escape'.  What I love about this is at a glance it can show me the time, if I am still on the VPN and if I have any notifications.  I can now walk in the room, check this and walk out again if no one is trying to grab my attention instead of unlocking to check this.
Note that the background above is an HP one I found long ago.  I thought it fitted in quite nicely.  When you get past this screen you see a password entry unlock screen.
There have been other improvements that are useful but not in-your-face.  A quick list I have found so far:
  • The printers section of System Settings can now configure the printer correctly
  • The mouse/touchpad settings screen appears to have been re-designed.  It has more settings but all of them appear to be useful
  • By popular demand there is now a "Power off" option by default instead of "Suspend".  Hitting the alt key shows "Suspend" instead so they have been switched.
  • WiFi connections are ordered by strength, not name! (many thanks for fixing that one guys)
Of course there are things I'm not so happy about yet (I haven't had time to try and like them):
  • Maximizing nautilus hides the title bar (not that I use it maximized often)
  • The clock still doesn't support multiple simultaneous timezones yet
These don't yet distract from the overall experience.  If I had to give a proper review I would so far give it 9/10.  Great work Gnome team.

Thursday, 4 October 2012

Openstack - Coding at Scale Keynote

This morning I gave a short keynote at O'Reilly's Velocity Europe entitled 'Openstack - Coding at Scale'

See the video below.

Tuesday, 24 July 2012

My journey so far

July marks my sixth month working on the OpenStack project for HP. Over the past few years I have had people ask me about my journey to where I am now, especially students working on the Drizzle project. So I decided to write it today.

Back in 2007 I worked as a freelancer developing PHP code, doing DBA work and administering Linux systems. One of my biggest clients was an online magazine called The First Post, I was doing so much work for them they hired me as a full time employee under the title Technical Architect. We made great strides whilst working there and I even got the site running from a MySQL Cluster installation.

Towards the end of 2007 the magazine was in financial difficulty and there was a real risk of everyone losing their jobs over Christmas. As luck would have it one of our biggest fans was a guy called Felix Dennis who owns a magazine empire called Dennis Publishing. They bought our magazine business and my responsibilities grew from maintaining one site to many. Dennis paid for me to get several certifications including MySQL DBA and developer.

One of my friends in the PHP community called Ligaya mentioned that Sun Microsystems who had recently purchased MySQL AB was hiring MySQL support engineers. This is a cross between a customer support role and developer role since we spent a lot of time debugging MySQL source code and suggesting patches to the developers. Spending lots of time in tools such as GDB was a fun challenge, especially on platforms such as AIX.

At one of the MySQL user conference I woke up very early on the first day, switched on the news to see that Oracle had agreed to buy Sun. I thought I was still dreaming but it turns out it was very real. What followed was several very difficult months for us, the regulators were trying to make sure everything was above board and every day we waited meant more job cuts and uncertainty from the public. We were censored from replying about what was happening which was very hard for a team of very vocal people. Eventually the sale was completed and things started to change.

Oracle had a very corporate attitude way to a lot of things which made things like getting paid correctly very difficult. Several of my friends had left or were leaving so I decided it was time for me to move on. A few friends had been hired by Rackspace to work on a radical fork of MySQL called Drizzle and they offered me a place there. It was a fantastic job, real open source the way it should be, the development was very rapid and well tested. Unfortunately it wasn't to last, a couple of days after we released the first GA of Drizzle we were told that Rackspace was no longer going to fund its development. This came shortly before the MySQL user conference where I was booked to give around 8 hours of talks.

I don't remember much of my talks at the conference but I'm sure they were poor. What I do remember was the stress and depression I felt at the time caused my hair to start falling out whilst I was at the conference. We had to give the impression that all was well at the conference despite knowing that it wasn't. I saw a good friend on the project go through similar things to me. It led in the end to me losing half my hair to alopecia and being on anti-depressants for the rest of the year.

Some of my friends that left Oracle had moved on to a company providing MySQL products and support called SkySQL. I joined them providing L3 support and developing new products. I was still going through my spell of depression at the time and came to the decision that I needed a complete fresh start away from MySQL and the 'community' around it.

Whilst on vacation in the USA I was called up to interview with HP's cloud department in Seattle. Working on real open source again sounded like a fantastic opportunity and I jumped at the chance. It has worked out great so far. I am getting my own team soon, am off the medication and my hair has grown back! I work on the developer continuous integration and tooling with many great people (several of whom have had similar journeys to mine).

I often get job offers to work on MySQL again but I think it very unlikely an adventure I would ever continue.

Tuesday, 29 May 2012

OpenStack's Jenkins Job Filler

OpenStack comprises of many projects, all of which have code reviews via. Gerrit and every code review is tested with many tests on Jenkins.  Whilst this is excellent for code quality this can get quite insane to manage.  In fact at time of writing OpenStack's Jenkins server now has a shade under 300 jobs to manage, I don't know about you but I really don't want to be spending my days managing all of that.  Often we need to make a small change to many jobs at the same time and human error occurs.  In the last few weeks we developed and unveiled the CI team's solution to this: Jenkins Job Filler (aka. Jenkins Job Builder when I forget what we called it).

The initial version of Jenkins Job Filler was written in a combination of Ruby and Puppet scripts but unfortunately it became unmanageable very quickly, it also required a Jenkins restart on every change.  So the new version has been writing in Python and YAML.  There is still some puppet behind it, but only to execute the Python script on every run.

How does it work?  There are two ways of specifying a project, one way is to have a YAML section in a file for every job, the other is to use a template.  With the template system this is all that is needed to specify all the jobs for Nova:

 template: 'python_jobs'

  name: 'nova'
  disabled: 'false'
  github_org: 'openstack'
  review_site: ''
  publisher_site: ''

Much simpler than hacking XML files!

Behind this there are python modules which specify each part of a standard XML file and the template YAML file which specifies which modules should be used for each job and the parameters for each job.  It then injects the resulting XML straight into Jenkins using the Jenkins API.

This system is currently in use for all of StackForge and most of OpenStack (the last few remaining jobs will be converted over the next couple of weeks).  Now if we want to make a change to one job in all projects we can do so in one go as a tiny commit.  This is made much simpler for OpenStack due to the standard Project Testing Interface that projects should follow.  But it is flexible enough to work with non-standard jobs and projects.

To find out more see the Jenkins Jobs Documentation or the code in the OpenStack CI Puppet repository.

Wednesday, 11 April 2012

Juniper VPN in Fedora

Whilst I am not a fan of Juniper Network Connect in the last few years I have had to connect to several networks that use it.  The biggest problem with it is that it is a combination of Java and a 32bit C library which will not work when executed with a 64bit version of Java.
There are other ways of connecting such as the Mad Scientist script but if you use things like two-factor authentication this will not work.  So I brought together things I have learnt from web postings about getting it to work in Ubuntu and have created these steps.  They work in Fedora 17 and should work in 16 too:

Step 1

We need to install OpenJDK Java, we also need xterm for the root password during installation:
sudo yum install java-1.7.0-openjdk.i686 java-1.7.0-openjdk.x86_64 icedtea-web xterm

Step 2

Now we need to tell Java to execute the Juniper binaries in 32bit mode but everything else in 64bit mode, to do this we first rename the 64bit Java binary:
cd /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/
mv java java.orig

Then create a replacement file called “java” with the following:
if [ $3x = "NCx" ]
    /usr/lib/jvm/jre-1.7.0-openjdk/bin/java "$@"
    /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java.orig "$@"

Finally make this executable:
sudo chmod +x java

Step 3

Log on to your VPN's website as normal, an xterm session will pop up for your sudo password the first time you use this to install it.

Wednesday, 21 March 2012

New Gerrit and Jenkins styles are live

After a bit of tweaking and improvements following feedback from various people the new changes to Jenkins and Gerrit are now live. The changes will also be rolled out onto Stackforge in the next 24 hours.

I've had a lot of great feedback in the few hours the changes have been live, many love it and some have suggestions for improvements. This is great, we know the look could use a bit of refining here and there. What makes this truly awesome is that the styles are kept in an OpenStack project with which anyone can file bugs or send patches up to Gerrit.

To modify the styles simply grab the openstack-ci-puppet repo and look in the Gerrit and Jenkins modules for the files. If you wish to file a bug, please do so in the OpenStack CI bugs page.

Thanks again for the great feedback so far. Hopefully this has made the lives of many of you that little bit easier (or at least stop your eyes bleeding).

Thursday, 15 March 2012

The real way to start hacking on OpenStack

I've seen recent attempts at blog posts to show how to get started at hacking on an OpenStack project.  Unfortunately they seem to have over-complicated the issue for new users.  As part of the Core Infrastructure team it is my job to make submitting, reviewing and merging code easier for developers.  This includes documenting that process, so here goes :)

  1. You need a Launchpad account and need to be joined to the Openstack team.  You can also join the team of one of the many subprojects if you want to.  Make sure Launchpad has your SSH key, Gerrit (the code review system) uses this.
  2. Sign the CLA as outlined in section 3 of the How To Contribute wiki page
  3. Tell git your details:
    git config --global "Firstname Lastname"
    git config --global "" 
  4. Install git-review. This tool takes a lot of the pain out of remembering commands to push code up to Gerrit for review and to pull it back down to edit it. It is installed using:
    pip install git-review
    Several Linux distributions (notably Fedora 16 and Ubuntu 12.04) are also starting to include git-review in their repositories so it can also be installed using the standard package manager.
  5. Grab a tree to hack on, for example for Nova you would do:
    git clone git://
  6. Checkout a new branch to hack on:
    git checkout -b TOPIC-BRANCH
  7. Start hacking
  8. Run the test suite locally to make sure nothing broke
  9. Commit your work using:
    git commit -a
    or you can use the following to edit a previous commit:
    git commit -a --amend
  10. Push the commit up for code review using:
    git review
    That is the awesome tool we installed earlier that does a lot of hard work for you
  11. Watch your email or review site, it will automatically send your code for a battery of tests on our Jenkins setup and the core team for the project will review your code. If there is any changes that should be made they will let you know.
  12. When all is good the review site will automatically merge your code
Obviously nearly half of that list are tasks you will only need to perform once. So as you can see it is pretty darn simple to get started. If anyone gets stuck they are welcome to shout out in the #openstack-dev IRC channel or the OpenStack mailing list. For more information on developer work flow please see the Gerrit Workflow page, I'm not ashamed to say I still have it as a pinned tab :)

Tuesday, 13 March 2012

Changes coming to Gerrit's style

Gerrit is a fantastic tool for performing code reviews and automating many of the tasks which can become quite complex. Unfortunately its default style is not the most attractive thing to look at all day. Yesterday I was tasked with skinning Gerrit to make it more in-line with Openstack. This was not as easy a task as I first anticipated, mostly down to not having the ability to edit the HTML layout very much without editing Gerrit's source.

The result isn't perfect but hopefully a notable improvement which will be coming to an Openstack review site near you soon.  Here is the before and after shots:

Wednesday, 18 January 2012

A Change in Direction

In 2008 my career took a sudden unexpected turn into the world of MySQL when I was offered a job at Sun.  Since then MySQL and it's forks have been a big part of my life.  The whole community (I mean the people, not the companies) around MySQL are part of what really drove me.

Unfortunately to me something has changed.  I am not exactly sure what it is, but I am sure it is not just me because others have expressed it in conversation too.  I wasn't enjoying things as much as I used to and for several reasons, some related to this, I have been quite ill.

Recently I was approached by HP's new cloud division who wanted me to work on OpenStack.  It seemed the perfect opportunity to start something new inside a new vibrant community.  That is not to say I have anything against my previous employer, SkySQL.  They are doing a fantastic job with a great vision and I wish them well in the future.

So, I have been working on OpenStack's Core Infrastructure team for a week and a half now and so far I am loving it.  The community is very welcoming and it is refreshing to see a development model similar to the one used in Drizzle so widely adopted.  I look forward to diving deeper into the OpenStack world and blogging about it as I go.