Libdrizzle Redux 5.0-alpha1 Released!

Over the past few months I have been spending my spare time on a new project.  A new version of libdrizzle which is much simpler to use and with many new features.  Today the first version of this MySQL compatible client is released, called Libdrizzle Redux.

Why 5.0?  Because Libdrizzle 1.0 and 2.0 have already been released, in packaging versions 3.0 and 4.0 used as API revisions.  So 5.0 is the fresh start.

Main Features

These are the main features of the library:

• A BSD licensed MySQL compatible C connector, so you can statically link it with commercial software
• A simplified API compared to Libdrizzle.  No more confusion over whether the client or library should be allocating/freeing.  There isn't a big difference to the MySQL C API for most things.
• New documentation, PDF for now web based coming soon.
• A new binlog reading API.  This can connect as a MySQL slave or mysqlbinlog client and get each event from the binary log.

Differences from Libdrizzle

Whilst based on Libdrizzle, Libdrizzle Redux has had an overhaul:

• The API is a little different from the older Libdrizzle library, for the most part it has been simplified
• The server-side API has been stripped out, this is now a client-only connector
• The new Binlog API
• Improved documentation

Still to Come

These is still a lot of work to do.  There are several features I want to get in over the next 6 months and improve the features we already have.  Some example of the improvements needed are:

• Improvements to documentation with better examples and a breakdown of the explanation of the examples and how they work
• More tests!  Testing against more platforms too
• A different build system.  I used CMake for now for RAD purposes, Brian Aker is already working on switching this out for an autotools based system.
• Server-side prepared statement support
• Several new client projects using this library

This is a fully open source project, so anyone can come along and hack on it.

Download

You can download it now from Launchpad along with the documentation.  The download page is here.

Slides for Connectors Talk

I could not find a way to upload my slides for today's talk entitled "MySQL Compatible Open Source Connectors" on the Percona Live website so the PDF can be viewed on Slideshare.  Enjoy!

Upcoming MySQL Conferences

I may not work on many MySQL related things any more but there are some MySQL ecosystem events that I am participating in.

First there is Percona Live London.  This event is on the 3rd and 4th of December at The Millennium Gloucester Conference Centre.  At this event I will be giving a talk on MySQL Compatible Open Source Connectors.  This will cover alternatives to the standard connectors and how to use them, it will be useful if you find the licensing behind the standard connectors prohibiting.  There is also a 40% off discount code you can use when registering, simply enter the promo code "Come2mytalk".

On the 2nd and 3rd of February is FOSDEM.  At this event there is a "MySQL and Friends" dev room on the Sunday.  I am on the committee overseeing the talks submitted to this room and am looking forward to some great talk submissions.  One of the great things about FOSDEM is it is completely free and no registration is required.

I'm looking forward to seeing everyone related to the MySQL ecosystem at these events!

A few days with Gnome 3.6

A few days ago I decided to upgrade one of my laptops to the current alpha (soon to be beta) of Fedora 18.  There have been a lot of changes to Gnome with the 3.6 release and there have been may critics of these changes.  I actually want to stand-up and say that I am a happy Gnome 3 user, I've used it since way before 3.0 was released and am impressed with some of the changes in 3.6.

Most of the things I like in Gnome 3.6 are subtle changes.  For example I am British but all my laptops come from the US with US keyboard layouts.  Whilst docked I connect a UK keyboard to them.  Gnome 3 supports multiple keyboards but I usually find that after I unlock the screen it has reverted back to the default layout.  Now not only does this not happen but the new settings for it make it easy to set key bindings to switch layouts.

I don't yet know whether this is a Gnome thing or a Fedora thing, but I spend most of my time connected to a work OpenVPN server.  The NetworkManager icon now shows 2 icons, one with WiFi (showing signal) and one showing that I'm connected to a VPN.

These also show on the lock screen.  Speaking of the lock screen, there is now a "swipe-to-unlock" type screen before entering your password.  If you are a heavy keyboard user like me this can easily be cleared by a press of 'enter' or 'escape'.  What I love about this is at a glance it can show me the time, if I am still on the VPN and if I have any notifications.  I can now walk in the room, check this and walk out again if no one is trying to grab my attention instead of unlocking to check this.

Note that the background above is an HP one I found long ago.  I thought it fitted in quite nicely.  When you get past this screen you see a password entry unlock screen.

There have been other improvements that are useful but not in-your-face.  A quick list I have found so far:

• The printers section of System Settings can now configure the printer correctly
• The mouse/touchpad settings screen appears to have been re-designed.  It has more settings but all of them appear to be useful
• By popular demand there is now a "Power off" option by default instead of "Suspend".  Hitting the alt key shows "Suspend" instead so they have been switched.
• WiFi connections are ordered by strength, not name! (many thanks for fixing that one guys)

Of course there are things I'm not so happy about yet (I haven't had time to try and like them):

• Maximizing nautilus hides the title bar (not that I use it maximized often)
• The clock still doesn't support multiple simultaneous timezones yet

These don't yet distract from the overall experience.  If I had to give a proper review I would so far give it 9/10.  Great work Gnome team.

Openstack - Coding at Scale Keynote

This morning I gave a short keynote at O'Reilly's Velocity Europe entitled 'Openstack - Coding at Scale'

See the video below.

My journey so far

July marks my sixth month working on the OpenStack project for HP. Over the past few years I have had people ask me about my journey to where I am now, especially students working on the Drizzle project. So I decided to write it today.

Back in 2007 I worked as a freelancer developing PHP code, doing DBA work and administering Linux systems. One of my biggest clients was an online magazine called The First Post, I was doing so much work for them they hired me as a full time employee under the title Technical Architect. We made great strides whilst working there and I even got the site running from a MySQL Cluster installation.

Towards the end of 2007 the magazine was in financial difficulty and there was a real risk of everyone losing their jobs over Christmas. As luck would have it one of our biggest fans was a guy called Felix Dennis who owns a magazine empire called Dennis Publishing. They bought our magazine business and my responsibilities grew from maintaining one site to many. Dennis paid for me to get several certifications including MySQL DBA and developer.

One of my friends in the PHP community called Ligaya mentioned that Sun Microsystems who had recently purchased MySQL AB was hiring MySQL support engineers. This is a cross between a customer support role and developer role since we spent a lot of time debugging MySQL source code and suggesting patches to the developers. Spending lots of time in tools such as GDB was a fun challenge, especially on platforms such as AIX.

At one of the MySQL user conference I woke up very early on the first day, switched on the news to see that Oracle had agreed to buy Sun. I thought I was still dreaming but it turns out it was very real. What followed was several very difficult months for us, the regulators were trying to make sure everything was above board and every day we waited meant more job cuts and uncertainty from the public. We were censored from replying about what was happening which was very hard for a team of very vocal people. Eventually the sale was completed and things started to change.

Oracle had a very corporate attitude way to a lot of things which made things like getting paid correctly very difficult. Several of my friends had left or were leaving so I decided it was time for me to move on. A few friends had been hired by Rackspace to work on a radical fork of MySQL called Drizzle and they offered me a place there. It was a fantastic job, real open source the way it should be, the development was very rapid and well tested. Unfortunately it wasn't to last, a couple of days after we released the first GA of Drizzle we were told that Rackspace was no longer going to fund its development. This came shortly before the MySQL user conference where I was booked to give around 8 hours of talks.

I don't remember much of my talks at the conference but I'm sure they were poor. What I do remember was the stress and depression I felt at the time caused my hair to start falling out whilst I was at the conference. We had to give the impression that all was well at the conference despite knowing that it wasn't. I saw a good friend on the project go through similar things to me. It led in the end to me losing half my hair to alopecia and being on anti-depressants for the rest of the year.

Some of my friends that left Oracle had moved on to a company providing MySQL products and support called SkySQL. I joined them providing L3 support and developing new products. I was still going through my spell of depression at the time and came to the decision that I needed a complete fresh start away from MySQL and the 'community' around it.

Whilst on vacation in the USA I was called up to interview with HP's cloud department in Seattle. Working on real open source again sounded like a fantastic opportunity and I jumped at the chance. It has worked out great so far. I am getting my own team soon, am off the medication and my hair has grown back! I work on the developer continuous integration and tooling with many great people (several of whom have had similar journeys to mine).

I often get job offers to work on MySQL again but I think it very unlikely an adventure I would ever continue.

OpenStack's Jenkins Job Filler

OpenStack comprises of many projects, all of which have code reviews via. Gerrit and every code review is tested with many tests on Jenkins.  Whilst this is excellent for code quality this can get quite insane to manage.  In fact at time of writing OpenStack's Jenkins server now has a shade under 300 jobs to manage, I don't know about you but I really don't want to be spending my days managing all of that.  Often we need to make a small change to many jobs at the same time and human error occurs.  In the last few weeks we developed and unveiled the CI team's solution to this: Jenkins Job Filler (aka. Jenkins Job Builder when I forget what we called it).

The initial version of Jenkins Job Filler was written in a combination of Ruby and Puppet scripts but unfortunately it became unmanageable very quickly, it also required a Jenkins restart on every change.  So the new version has been writing in Python and YAML.  There is still some puppet behind it, but only to execute the Python script on every run.

How does it work?  There are two ways of specifying a project, one way is to have a YAML section in a file for every job, the other is to use a template.  With the template system this is all that is needed to specify all the jobs for Nova:

project:
 template: 'python_jobs'

values:
  name: 'nova'
  disabled: 'false'
  github_org: 'openstack'
  review_site: 'review.openstack.org'
  publisher_site: 'nova.openstack.org'

Much simpler than hacking XML files!

Behind this there are python modules which specify each part of a standard XML file and the template YAML file which specifies which modules should be used for each job and the parameters for each job.  It then injects the resulting XML straight into Jenkins using the Jenkins API.

This system is currently in use for all of StackForge and most of OpenStack (the last few remaining jobs will be converted over the next couple of weeks).  Now if we want to make a change to one job in all projects we can do so in one go as a tiny commit.  This is made much simpler for OpenStack due to the standard Project Testing Interface that projects should follow.  But it is flexible enough to work with non-standard jobs and projects.

To find out more see the Jenkins Jobs Documentation or the code in the OpenStack CI Puppet repository.