The importance of backup verification

I have recently moved to HP's Advanced Technology Group which is a new group in HP and as part of that I will be blogging a lot more about the Open Source things I and others in HP work on day to day.  I thought I would kick this off by talking about work that a colleague of mine, Patrick Crews, worked on several months ago.

For those who don't know Patrick, he is a great Devops Engineer and QA.  He will find new automated ways of breaking things that will torture applications (and the Engineers who write them). I don't know if I am proud or ashamed to say he has found many bugs in code that I have written by doing the software equivalent of beating it with a sledgehammer.

Every Devops Engineer worth his salt knows that backups are important, but one thing that is regularly forgotten about is to check whether the backups are good.  A colleague of mine from several years back, Ian Anderson, once told me about the hunt for a good tape archive vendor.  He tested them by getting them to pick a randomly selected tape from the archives and reading it, timing how long it takes to do so.  You would be surprised the vendors who couldn't perform this task, I'd hate to see what would happen in a real emergency.

There was also the case of CouchSurfing which was crippled back in 2006 when after a massive failure they found their backups to be bad.  They eventually rebuilt and is a great site today, but this kind of damage can cost even a small company many thousands of dollars.

The main thing I am trying to stress here is that it is important not just to make backups but also to make sure the backups are recoverable.  This should be done with verification and even fire drills.  There may come a time where you may really need that backup in an emergency and if it isn't there, well you just burnt your house down.

Before I was a member of the Advanced Technology Group I was the Technical Lead for the Load Balancer as a Service project for HP Cloud.  We had a very small team and needed a backup and verification solution that was mostly automated and reliable.  This thing would be hooked up to our paging system and none of my team like being woken at 2AM :)

This is a very crude diagram of the solution Patrick developed:

The solution works as follows:

1. Every X minutes Jenkins will tell the database servers to make a backup
2. The MySQL database servers encrypt that backup and push it up to the cloud file storage (Openstack Swift)
3. Jenkins would then trigger a compute instance (Openstack Nova) build and install MySQL on it
4. The new virtual machine would grab the backup, decrypt it, restore it and run a bunch of tests on the data to see if it is valid
5. If any of the above steps failed send out a page to people

Most of the above uses salt to communicate across the machines.  Have we ever been paged by this system?  Yes, but so far only because step 3 failed, either due to a Nova build fail or once due to a salt version incompatibility.  We have since added some resilience into the system to stop this happening again.

As well as the above there are monthly fire drills to manually test that we can restore from these backups.  We also regularly review the testing procedures to see if we can improve them.

This is going to sound a little strange but sometimes the best Devops Engineers are lazy.  By that I don't mean they don't do any work (they are some of the hardest working people I know), but they will automate everything so they don't have to do a lot of boring manual labour and they hate being woken by pagers at silly hours of the morning.  Some of the best Devops Engineers I know think in these terms :)

Stackforge Libra - Balance your life!

I have been pretty quiet on the blogging front for quite a long time now.  The main reason for this is I have been working very hard on leading a small team which is developing a Stackforge project called 'Libra'.  As you can probably guess from the name, Libra is a Load Balancer as a Service system.  Many of you may not of heard of it but according to Stackalytics it was the 27th biggest project in terms of code contributed during Havana and 2nd biggest in HP (something I am especially proud of because it has been one of the smallest teams in HP Cloud).

It is based on the Atlas API specifications, creates software-based load balancers, is implemented in Python and sits on top of Nova instances rather than working under the cloud.  It also has several unique features which could be converted to run any service on top of a cloud.

Components

Libra consists of several components, the service components are designed to be installed on multiple instances to create a highly available setup:

• API server - A Pecan based API server whose API is based on the Atlas API spec.
• Admin API server - A Pecan based Administrative API server (work in progress) and a whole bunch of modules which automatically maintain the health of the Libra system.
• Pool Manager - A gearman service which will provide the rest of the system with load balancer devices and floating IPs
• Worker - A gearman service sitting on each load balancer to configure it

There is also an easy-to-use command line client called python-libraclient.

The service requires Gearman to communicate between the various components and MySQL as a load balancer configuration data store. Our team has been developing SSL support in Gearman to increase the security of the communications and Libra is probably the first open source project to support this.  Security is a high priority for HP and the Libra team take it very seriously. Libra also supports Galera clusters for MySQL by having built-in auto-failover and recovery as well as commit deadlock detection and transaction retry.  The codebase can support any load balancing software (and in theory hardware) through plugins to the Worker but currently HAProxy is the only plugin being developed.

Auto-Recovery

Libra also has an intelligent auto-recovery system built in. In HP we are currently testing the 4.x release in a few racks that have a flaky networking setup and it has been consistently repairing devices just like a T-1000 repairs bullet holes.  Somehow we have even been able to create usable load balancers from this setup too!

The system is designed to keep a constant pool of hot spare load balancers on standby ready so that one can be provisioned very quickly.  The health of this pool is constantly checked, bad devices in the pool are destroyed and replaces automatically with new ones.

With version 3.x (HP Cloud Private Beta) we are seeing provision times of roughly 300ms.  This has increased slightly for Libra 4.x because it introduces floating IP support (called Virtual IPs in Atlas API).  The floating IP support means that the Libra system can automatically detect when a load balancer has failed, within seconds rebuild the load balancer on a new hot spare and move the IP address accordingly.

Other Uses

The architectural design of Libra is such that it could be used to create any "Platform-as-a-Service" on top of Nova (or any other cloud with minor modifications).  The system can be modified to work with anything by changing the API and giving it a Worker plugin.  This gives Libra codebase the ability to become an eventual framework in the future.

Libra and HP

Libra 3.x is currently installed as our LBaaS Private Beta and an installation of 4.x will be coming in the next few weeks.  We have learnt a lot from customer feedback and have a lot of interesting ideas in the pipeline or in testing (such as Galera cluster load balancers!  Handy for DBaaS).  If you want to find out more about Libra you can see the HP Cloud LBaaS page, look at the developer docs, come chat to us on #stackforge-libra Freenode IRC channel and if you want to give it a spin take a look at our PPA.

We want to be more open with the development process of Libra and are taking steps to ensure that happens by engaging more with the wider community, resource constraints have made this difficult up until now.  We are happy to help anyone who wants to get started playing with or hacking on any part of it.

Libdrizzle 5.1.4 released!

I've just created a release of Libdrizzle 5.1.4 (the BSD licensed C connector for MySQL servers) which for me is one of the most exciting releases to date.

Why?
Not because the amount of features or anything that we have added, in fact this is mostly a minor release with bug fixes.  It is because Brian Aker and I didn't submit the most code for it.  That honour goes to Wim Lewis from The Omni Group who has done a fantastic job fixing up Libdrizzle.

As for the release itself the main fixes revolve around cleaning up code and many fixes to the server-side prepared statement handling.  Behind the scenes Wim has supplied many improvements to the test suite and Brian has setup more platforms to test against in Jenkins.  All this to help improve the code quality of Libdrizzle.

You can download this new release along with older releases from the Libdrizzle downloads page.

MySQL and the forks in the road

There's a lot of activity in the MySQL ecosystem currently. I've written an article for The H which takes a look at MySQL and the various forks and spinoffs to give an overview of where MySQL's community and close competition is currently at.

http://www.h-online.com/open/features/MySQL-and-the-forks-in-the-road-1829242.html

First version of Drizzle Tools for MySQL servers released

Today marks the first release of Drizzle Tools for MySQL servers.  Drizzle Tools aims to be a collection of useful utilities to use with MySQL servers based around the work on the Libdrizzle Redux project.

In this first version there is one utility in the tree called 'drizzle-binlogs'.  If you've seen me talk about this tool before it is because it used to be included in the Libdrizzle 5.1 source but has now been moved here to be developed independently.  For those who haven't 'drizzle-binlogs' is a tool which connects to a MySQL server as a slave, retrieves the binary log files and stores them locally.  This could be used as part of a backup solution or a rapid way to help create a new MySQL master server.

Due to the API changes before the Libdrizzle API became stable Drizzle Tools requires a minimum of Libdrizzle 5.1.3 to be installed.

I wanted to release this sooner but unfortunately most of my time has been taken up with the first release of the project I manage and develop for my day job (HP Cloud's Load Balancer as a Service, more about this in a future blog post).

In the not too distant future there will be more tools included in the Drizzle Tools releases, I have the next one already 50% developed.  In the mean time you can download the first version here.

Using the Libdrizzle Binlog API

Now that we have frozen the 5.1 API of Libdrizzle I can blog about how to use parts of the API.

In this blog post I will cover connecting to a MySQL server and retrieving the binary logs.

First of all we need to connect to the MySQL server

#include <libdrizzle-5.1/libdrizzle.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <inttypes.h>

int main(void)
{
  drizzle_st *con;
  drizzle_binlog_st *binlog;
  drizzle_return_t ret;
 
  con= drizzle_create("localhost", 3306, "user", "pass", "", NULL);
  ret= drizzle_connect(con);
  if (ret != DRIZZLE_RETURN_OK)
  {
    printf("Could not connect to server: %s\n", drizzle_error(con));
    return EXIT_FAILURE;
  }

We are connecting to a local server on port 3306 with the username "user" and password "pass".  We don't need a default database so we leave that empty and set the options to NULL.

After we are connected to the MySQL server we need to initialize the binary log transfer.

  binlog= drizzle_binlog_init(con, binlog_event, binlog_error, NULL, true);

This states that we want an event to call function called "binlog_event" and an error to call a function called "binlog_error".  We could pass a pointer data to callback functions but for this example we have set it to "NULL".  The final parameter specifies whether or not we want to verify the checksums in MySQL 5.6 binary logs.

Then we need to start the binary log transfer.

  ret= drizzle_binlog_start(binlog, 0, "", 0);
  if (ret != DRIZZLE_RETURN_EOF)
  {
    return EXIT_FAILURE;  }
  drizzle_con_quit(con);
  return EXIT_SUCCESS;
}

This states that we want to start the binary log at the first position of the first file using server ID 0.  With server ID 0 the MySQL server will end the transfer at the end of the last binary log file.  The function will return at the end of the transfer with DRIZZLE_RETURN_EOF or an error code.

All we need now are error callback and event callback.  So lets start with the error callback.

void binlog_error(drizzle_return_t ret, drizzle_st *con, void *context)
{
  (void) context;
  if (ret != DRIZZLE_RETURN_EOF)
  {
    printf("Error retrieving binlog: %s\n", drizzle_error(con));
  }
}

Finally we need to event callback.

void binlog_event(drizzle_binlog_event_st *event, void *context)
{
  (void) context;
  uint32_t length;
  printf("Timestamp: %"PRIu32"\n", drizzle_binlog_event_timestamp(event));
  printf("Type: %"PRIu8"\n", drizzle_binlog_event_type(event));
  printf("Server-id: %"PRIu32"\n", drizzle_binlog_event_server_id(event));
  printf("Next-pos: %"PRIu32"\n", drizzle_binlog_event_next_pos(event));
  length= drizzle_binlog_event_length(event);
  printf("Length: %"PRIu32"\n", length);
  data= drizzle_binlog_event_data(event);
  printf("Data: 0x");
  for (i=0; i<length; i++)
    printf("%02X ", data[i]);
  printf("\n\n");
}

On every event received this function is called and will basically spit out the event details.

For a further example see the source of the drizzle_binlogs tool in the Drizzle Tools tree.

Libdrizzle 5.1.3 released

A couple of days ago we released Libdrizzle 5.1.3.  With this release of the C connector for MySQL servers we are freezing the 5.1 API and declaring it stable.  This is also one of our biggest releases after incorporating code from a Seattle developer day.  The diff since 5.1.2 is over 6000 lines long and around 180KBytes, incorporating many bug fixes and improvements.

The most notable changes in this release are:

• the drizzle_binlogs tool has been removed, it is now in the Drizzle Tools tree which will have its first release soon.
• the connection API has been refactored, options processing has been re-written and the connection API has been simplified in general
• drizzle_escape_string has been made safer
• drizzle_hex_string and drizzle_mysql_password_hash has been removed
• internal state system refactored to be safer
• binlog API changed to use a callback based API
• general API cleanups

To download the source and documentation of the latest release simply go to the download page on Launchpad.

Developing Libdrizzle

This weekend I am supposed to be giving a talk at FOSDEM on Libdrizzle.  Unfortunately my kids and I all fell ill on Thursday (my wife appears to be immune) so I had to cancel my plans (infecting 5000 people didn't seem wise :)

Instead I am writing this blog post about Libdrizzle and my part in it which covers some of what I was going to talk about.

History of Libdrizzle

Libdrizzle started out as a from-scratch C connector for Drizzle and MySQL originally created by Eric Day.  It was designed to be high performance and use common standards to make it easy to work on.  In the summer of 2010 it was merged into the main Drizzle tree where development has been focused.  There were several attempts to split it out again but until now none were truly successful.

For a few years Brian Aker and I have been randomly discussing things we could have done to make the API easier for developers and users.  As well as features we could possibly add in the future.  Fast forward to Summer 2012 and I decided to do something about this.

The Dawn of Libdrizzle Redux

Around July 2012 I took the Libdrizzle 1.0 code from Drizzle trunk and got it to compile separately using CMake.  I then had the groundwork to work on some of the things we had talked about.  I called this the Libdrizzle Redux project since we were bringing new life to Libdrizzle, primarily as a MySQL Connector.

Initially this was going to be version 3.0 but it was pointed out by Henrik that 3.0 and 4.0 were used as library versions in certain packages so we went straight to 5.0.

After getting to the "It Compiles!" state the first thing I did was strip out the server side API.  I had spoken to many people over the years about the connector and each one only used the client side API, not the server side.  The other thing that was made plainly clear was the API was too complex.  There were many functions that were confusing as to whether the library or the application was supposed to allocate and free objects.  So this was massively simplified very quickly.

Libdrizzle Today

Between Brian and I there have been massive changes to the Libdrizzle code and we now have the basis for a regression suite which has already found several problems with the original code.  Just off the top of my head:

• There is a new Prepared Statement API
• There is a new Binary Log Retrieval API (in 5.1.3 this will become a callback API) which is already MySQL 5.6 checksum compatible
• Massively simplified API, the number of things needed to setup a connection and execute a query have been reduced
• Non-blocking API has been made more stable
• Windows support (via. MinGW)
• Buffer and state system massively improved
• CMake has been replace with DDM4
• We compile in C++ with many warnings switch on (C++ compiler tends to find more problems than the normal C compiler)
• 106 revisions/merges in the BZR repo.  Most of this since November 2012.

Last Sunday Brian and I had a hacking day in Seattle, we worked non-stop on Libdrizzle for nearly 12 hours, I doubt there were many sections of code that were not touched that day.

Why I Develop Libdrizzle

My day job is the Project Manager on Load Balancer as a Service at HP Cloud (you should hear a lot more about that in coming months).  Whilst this is great I don't get to touch and debug a whole lot of code any more.  So Libdrizzle is something I do in my spare time to keep my brain fully active.

I also like to think that the code will be useful to someone.  I'm actually a fan of the BSD license in this context.  It means commercial code can easily statically compile with Libdrizzle which in-turn means more MySQL Server/Percona Server/MariaDB Server adoption and more support contracts for Oracle/Percona/SkySQL/etc...

Even if no one uses the code I still find it a fun challenge and something I will continue for as long as I can.

The Future of Libdrizzle

In the next few days we will be releasing Libdrizzle 5.1.3, this will contain all the work that was done in Seattle (and my flights to/from Seattle) including a few nice API improvements.  Along with this I will create the first release of Drizzle Tools.

Drizzle Tools is a project which contains utilities for use with a MySQL server, at the moment the only one included is drizzle_binlogs which connects to a remote MySQL server as a slave and downloads the binary logs to local files as a back.  It also has a continuous mode which effectively gives an up-to-the-second backup of the data.  I have the code half-written for the next tool which will not make the first release but should be useful to many people.

Of course anyone is welcome to come hack on Librizzle and Drizzle Tools, file bugs, ask questions, etc...  The more knowledge share the better :)

There will be more posts soon showing how to use Libdrizzle and drizzle_tools as well as announcing the new features, so watch this space!

Introducing Drizzle Tools

As part of the Libdrizzle Redux project I created an example tool which was bundled with it which will connect to a MySQL server as a slave and download the binary logs to local files.  This was developed as a quick example of what can be done with the new binlog API.

Two things quickly became apparent:

1. We shouldn't really be distributing applications in a library
2. I am going to be developing more useful tools around libdrizzle and they certainly shouldn't be in the same package
3. BSD is a fantastic license for a library, but I personally prefer GPLv2 for applications

With this in mind I have pulled the drizzle_binlogs utility from Libdrizzle trunk (and therefore won't be in the 5.1.3 release) and put it in its own repository.  It has been licensed appropriately (GPLv2) and I am already beginning to develop more tools to go with it.

The are no source tarballs and PDFs of the docs for Drizzle Tools yet, but you can get the bzr tree to play with.  The Launchpad page is here.  It requires Libdrizzle 5.1.2 to build against.

Some really cool stuff coming to that tree soon which I will announce when ready :)

Enjoy!

Libdrizzle 5.1.2 released and Libdrizzle at FOSDEM

Libdrizzle is getting better and better with every release, and to reflect this we have two announcements to make.

First of all, Libdrizzle 5.1.2 has been released.  This release has several major bug fixes and performance enhancements.  Changes are as follows:

• Non-blocking Windows connections are now more stable
• Improvements to Windows building
• Unix Socket connections are now more stable
• Memory allocation/freeing cleanups (for performance and ease of code)
• Network packet buffer now much more flexible
• Many performance improvements (the bundled drizzle_binlogs tool is now around 10x faster on my i7 laptop)
• Several other minor bug fixes

API chages:

• drizzle_query_str() has been removed, drizzle_query() with a 0 byte length parameter now does the same thing.

As always you can download the tarball and documentation from the Libdrizzle Launchpad Page.

I'm also going to be giving a talk entitled "Libdrizzle Redux, the next-gen MySQL compatible connector" on Sunday 3rd Feb in the MySQL devroom at FOSDEM.  This will be a great chance to find out more about the work we have been doing on the connector and how it can benefit you.

Libdrizzle 5.1.1 released!

A new version of the BSD licensed, MySQL compatible C client library, called Libdrizzle (5.1.1 codename Carbrooke) has been released today.

Since the last release (less than 2 weeks ago) the diff is 7725 lines long, 224KB in size.  So we have been really busy :)

There have been a few API changes in this version, especially around the prepared statement API.  We have made this API much simpler to use.  So applications will very likely need slight modifications and recompiling.

Major changes in this version:

• Windows support (using MinGW) - Big thanks to Brian Aker, he worked very hard on this
• More cleanups and improvements to the general API
• Faster, more stable non-blocking connection handling - Something else Brian has been working hard on
• Almost the entire prepared statement user API has been re-written.  It now uses get/set functions for the statement data.  It can also use column names instead of IDs to get the data
• Binary log functions now support MySQL 5.6's checksums (and is aware of the new event types)
• The glib2 dependency which was used for the drizzle_binlogs example utility has been removed
• Every branch merge to trunk for this release was tested on Jenkins!
• Many other bug fixes

You can download the source and documentation on the Launchpad downloads page.  We are aiming to have binary releases soon (although this may be around version 5.1.2).

We are still working on some awesome improvements and I intend to blog some examples soon.  Feel free to give at try, file a bug, ask a question and help us make this an even more awesome product :)

Questions about the MariaDB C Connector

I should first start this blog post with the acronym IANAL (I am not a lawyer).  Also the views in this post are mine and not my employer's (a lawyer did ask me to say that part).

Questions on the MariaDB JDBC Driver have already been raised so I wanted to do something similar for the MariaDB C Connector.  The JDBC's issues are mostly ethical whereas my C Connector questions are mostly legal.

But first a little history...  MySQL 3.23.58 was the last version to include an LGPL licensed connector for MySQL, the connector went GPL (and of course commercial) after this version.  Nowadays the connector is GPL with a FLOSS exception, that exception allows you to compile with certain other licensed software.  The main reason for this is PHP version 4 switched from a dual-licensed under the GPL to PHP licensed only.  The PHP license is incompatible with GPL and LGPL licensed in both directions, this is because clauses 3&4 of the PHP license conflicts with clause 6 of the GPL license (which is also clause 10 of the LGPLv2 license).  The FLOSS exception isn't in 3.23.58, it wasn't needed back then.

Why is this relevant?  The MariaDB C Connector is based on the LGPL 3.23 connector and is coded-up to have the current features of the MySQL connector.  Some of this is based on the PHP licensed MySQLnd connector.

Being LGPL licensed is one of the main selling points of the connector.  It means you can dynamically link the library with a commercial application.  Something you would have to buy a license for to do with the MySQL connector.

So, my questions are:

1. How can MariaDB's C Connector legally include LGPL and PHP licensed code in the same source under the LGPL umbrella?

2. There are questions around the licensing and availability of the native LGPL SDBC (OpenOffice) driver that other parts of the connector are based on.  In the comments of another blog post it was claimed it that the SDBC code was "... developed by MySQL in 2006 and 2007 and never reached GA – discontinued when Sun Microsystems acquired MySQL AB" but I haven't seen any evidence of this.  The only Sun code I can find for SDBC is 2008/2009 and LGPLv3 which can't be re-licensed as LGPLv2.  Now there is a newer MySQL SDBC driver which is Apache2 licensed.  Where can we find the code MaraDB based their work on?

3. Although not distributed in the packages there are GPL licensed test cases in the bzr source.  Can you globally call it LGPL with these in there?

I do hope answers to these can be put to rest so that customers can continue to be happy using the alternative connector.

A MySQL Christmas present - Libdrizzle 5.1.0

Brian Aker and I have been working hard in the last few weeks to give you a great Christmas present, Libdrizzle 5.1.0.  The MySQL compatible, BSD licensed C connector (so static compiling with commercial software gets the thumbs up!).

The latest changes include:

• A server-side prepared statement API
• Improved binary log API
• An example binary log remote retrieval utility using the binlog API called "drizzle_binlogs"
• A new build system, DDM4 which is used by Gearman and Memcached
• Many bugs fixes

The source and manuals can be found on the Launchpad downloads page.  Please enjoy, feel free to file bugs, questions and hack on code on our Launchpad page.  Happy holidays to all!

Libdrizzle Redux 5.0-alpha1 Released!

Over the past few months I have been spending my spare time on a new project.  A new version of libdrizzle which is much simpler to use and with many new features.  Today the first version of this MySQL compatible client is released, called Libdrizzle Redux.

Why 5.0?  Because Libdrizzle 1.0 and 2.0 have already been released, in packaging versions 3.0 and 4.0 used as API revisions.  So 5.0 is the fresh start.

Main Features

These are the main features of the library:

• A BSD licensed MySQL compatible C connector, so you can statically link it with commercial software
• A simplified API compared to Libdrizzle.  No more confusion over whether the client or library should be allocating/freeing.  There isn't a big difference to the MySQL C API for most things.
• New documentation, PDF for now web based coming soon.
• A new binlog reading API.  This can connect as a MySQL slave or mysqlbinlog client and get each event from the binary log.

Differences from Libdrizzle

Whilst based on Libdrizzle, Libdrizzle Redux has had an overhaul:

• The API is a little different from the older Libdrizzle library, for the most part it has been simplified
• The server-side API has been stripped out, this is now a client-only connector
• The new Binlog API
• Improved documentation

Still to Come

These is still a lot of work to do.  There are several features I want to get in over the next 6 months and improve the features we already have.  Some example of the improvements needed are:

• Improvements to documentation with better examples and a breakdown of the explanation of the examples and how they work
• More tests!  Testing against more platforms too
• A different build system.  I used CMake for now for RAD purposes, Brian Aker is already working on switching this out for an autotools based system.
• Server-side prepared statement support
• Several new client projects using this library

This is a fully open source project, so anyone can come along and hack on it.

Download

You can download it now from Launchpad along with the documentation.  The download page is here.

Slides for Connectors Talk

I could not find a way to upload my slides for today's talk entitled "MySQL Compatible Open Source Connectors" on the Percona Live website so the PDF can be viewed on Slideshare.  Enjoy!

Upcoming MySQL Conferences

I may not work on many MySQL related things any more but there are some MySQL ecosystem events that I am participating in.

First there is Percona Live London.  This event is on the 3rd and 4th of December at The Millennium Gloucester Conference Centre.  At this event I will be giving a talk on MySQL Compatible Open Source Connectors.  This will cover alternatives to the standard connectors and how to use them, it will be useful if you find the licensing behind the standard connectors prohibiting.  There is also a 40% off discount code you can use when registering, simply enter the promo code "Come2mytalk".

On the 2nd and 3rd of February is FOSDEM.  At this event there is a "MySQL and Friends" dev room on the Sunday.  I am on the committee overseeing the talks submitted to this room and am looking forward to some great talk submissions.  One of the great things about FOSDEM is it is completely free and no registration is required.

I'm looking forward to seeing everyone related to the MySQL ecosystem at these events!

My journey so far

July marks my sixth month working on the OpenStack project for HP. Over the past few years I have had people ask me about my journey to where I am now, especially students working on the Drizzle project. So I decided to write it today.

Back in 2007 I worked as a freelancer developing PHP code, doing DBA work and administering Linux systems. One of my biggest clients was an online magazine called The First Post, I was doing so much work for them they hired me as a full time employee under the title Technical Architect. We made great strides whilst working there and I even got the site running from a MySQL Cluster installation.

Towards the end of 2007 the magazine was in financial difficulty and there was a real risk of everyone losing their jobs over Christmas. As luck would have it one of our biggest fans was a guy called Felix Dennis who owns a magazine empire called Dennis Publishing. They bought our magazine business and my responsibilities grew from maintaining one site to many. Dennis paid for me to get several certifications including MySQL DBA and developer.

One of my friends in the PHP community called Ligaya mentioned that Sun Microsystems who had recently purchased MySQL AB was hiring MySQL support engineers. This is a cross between a customer support role and developer role since we spent a lot of time debugging MySQL source code and suggesting patches to the developers. Spending lots of time in tools such as GDB was a fun challenge, especially on platforms such as AIX.

At one of the MySQL user conference I woke up very early on the first day, switched on the news to see that Oracle had agreed to buy Sun. I thought I was still dreaming but it turns out it was very real. What followed was several very difficult months for us, the regulators were trying to make sure everything was above board and every day we waited meant more job cuts and uncertainty from the public. We were censored from replying about what was happening which was very hard for a team of very vocal people. Eventually the sale was completed and things started to change.

Oracle had a very corporate attitude way to a lot of things which made things like getting paid correctly very difficult. Several of my friends had left or were leaving so I decided it was time for me to move on. A few friends had been hired by Rackspace to work on a radical fork of MySQL called Drizzle and they offered me a place there. It was a fantastic job, real open source the way it should be, the development was very rapid and well tested. Unfortunately it wasn't to last, a couple of days after we released the first GA of Drizzle we were told that Rackspace was no longer going to fund its development. This came shortly before the MySQL user conference where I was booked to give around 8 hours of talks.

I don't remember much of my talks at the conference but I'm sure they were poor. What I do remember was the stress and depression I felt at the time caused my hair to start falling out whilst I was at the conference. We had to give the impression that all was well at the conference despite knowing that it wasn't. I saw a good friend on the project go through similar things to me. It led in the end to me losing half my hair to alopecia and being on anti-depressants for the rest of the year.

Some of my friends that left Oracle had moved on to a company providing MySQL products and support called SkySQL. I joined them providing L3 support and developing new products. I was still going through my spell of depression at the time and came to the decision that I needed a complete fresh start away from MySQL and the 'community' around it.

Whilst on vacation in the USA I was called up to interview with HP's cloud department in Seattle. Working on real open source again sounded like a fantastic opportunity and I jumped at the chance. It has worked out great so far. I am getting my own team soon, am off the medication and my hair has grown back! I work on the developer continuous integration and tooling with many great people (several of whom have had similar journeys to mine).

I often get job offers to work on MySQL again but I think it very unlikely an adventure I would ever continue.

A Change in Direction

In 2008 my career took a sudden unexpected turn into the world of MySQL when I was offered a job at Sun.  Since then MySQL and it's forks have been a big part of my life.  The whole community (I mean the people, not the companies) around MySQL are part of what really drove me.

Unfortunately to me something has changed.  I am not exactly sure what it is, but I am sure it is not just me because others have expressed it in conversation too.  I wasn't enjoying things as much as I used to and for several reasons, some related to this, I have been quite ill.

Recently I was approached by HP's new cloud division who wanted me to work on OpenStack.  It seemed the perfect opportunity to start something new inside a new vibrant community.  That is not to say I have anything against my previous employer, SkySQL.  They are doing a fantastic job with a great vision and I wish them well in the future.

So, I have been working on OpenStack's Core Infrastructure team for a week and a half now and so far I am loving it.  The community is very welcoming and it is refreshing to see a development model similar to the one used in Drizzle so widely adopted.  I look forward to diving deeper into the OpenStack world and blogging about it as I go.

Mydumper 0.5.1 released

After much feedback from the community who have been using mydumper I have created the first bugfix release of the 0.5 series of mydumper.  Changes in this release are as follows:

• Fix MySQL 5.0 compile issue


• Make the metadata file visible (after muliple requests)


• Add --no-lock option to mydumper


• First --daemon snapshot is now at execution instead of the first timed interval


• Fix CentOS 5.x compile issue (also affected Mac OSX)


• Fix libmysqlclient search bug


• Add cppcheck target


• Fix errors flaged by cppcheck


• Add option to turn off docs build


• Add status output at end of CMake

To obtain this release you can download it from Launchpad.

Many thanks to everyone who has been trying it and giving feedback.

An apology to MySQL@Oracle

About a week ago I appeared to have started an argument which has now spanned several blog post comments, 39 Google+ comments and two other blog posts.  So the first thing I want to do is apologise to the MySQL@Oracle staff (and Sheeri).  I was disagreeing on the use of 'The' whilst the real problem is my personal understanding the meaning of the word/name/trademark 'MySQL'.  You guys are right and I am wrong.  To correct this I'll refer to what I used to call the "MySQL Ecosystem" as the Open Database Community until someone has a cool name for it.

The arguments were so severe that last week I was even thinking about leaving the Open Database Community.  Apart from Drizzle, mydumper and a few other things I don't contribute as much useful content I used to now anyway.  I would need about another 20 hours in every day to do as much as I would like to do.  But for now I will just stay more in the background just pushing out code to anyone who wants it.

In reply to Herik's post, please don't blame Monty Taylor, this was all my fault.  Also I never intended for the argument to be hidden (or exist at all) on Google+.  Someone at Oracle pointed out that it was wrong to make any of those comments public, so they will stay hidden for now.