Developing Libdrizzle

This weekend I am supposed to be giving a talk at FOSDEM on Libdrizzle.  Unfortunately my kids and I all fell ill on Thursday (my wife appears to be immune) so I had to cancel my plans (infecting 5000 people didn't seem wise :)

Instead I am writing this blog post about Libdrizzle and my part in it which covers some of what I was going to talk about.

History of Libdrizzle

Libdrizzle started out as a from-scratch C connector for Drizzle and MySQL originally created by Eric Day.  It was designed to be high performance and use common standards to make it easy to work on.  In the summer of 2010 it was merged into the main Drizzle tree where development has been focused.  There were several attempts to split it out again but until now none were truly successful.

For a few years Brian Aker and I have been randomly discussing things we could have done to make the API easier for developers and users.  As well as features we could possibly add in the future.  Fast forward to Summer 2012 and I decided to do something about this.

The Dawn of Libdrizzle Redux

Around July 2012 I took the Libdrizzle 1.0 code from Drizzle trunk and got it to compile separately using CMake.  I then had the groundwork to work on some of the things we had talked about.  I called this the Libdrizzle Redux project since we were bringing new life to Libdrizzle, primarily as a MySQL Connector.

Initially this was going to be version 3.0 but it was pointed out by Henrik that 3.0 and 4.0 were used as library versions in certain packages so we went straight to 5.0.

After getting to the "It Compiles!" state the first thing I did was strip out the server side API.  I had spoken to many people over the years about the connector and each one only used the client side API, not the server side.  The other thing that was made plainly clear was the API was too complex.  There were many functions that were confusing as to whether the library or the application was supposed to allocate and free objects.  So this was massively simplified very quickly.

Libdrizzle Today

Between Brian and I there have been massive changes to the Libdrizzle code and we now have the basis for a regression suite which has already found several problems with the original code.  Just off the top of my head:

• There is a new Prepared Statement API
• There is a new Binary Log Retrieval API (in 5.1.3 this will become a callback API) which is already MySQL 5.6 checksum compatible
• Massively simplified API, the number of things needed to setup a connection and execute a query have been reduced
• Non-blocking API has been made more stable
• Windows support (via. MinGW)
• Buffer and state system massively improved
• CMake has been replace with DDM4
• We compile in C++ with many warnings switch on (C++ compiler tends to find more problems than the normal C compiler)
• 106 revisions/merges in the BZR repo.  Most of this since November 2012.

Last Sunday Brian and I had a hacking day in Seattle, we worked non-stop on Libdrizzle for nearly 12 hours, I doubt there were many sections of code that were not touched that day.

Why I Develop Libdrizzle

My day job is the Project Manager on Load Balancer as a Service at HP Cloud (you should hear a lot more about that in coming months).  Whilst this is great I don't get to touch and debug a whole lot of code any more.  So Libdrizzle is something I do in my spare time to keep my brain fully active.

I also like to think that the code will be useful to someone.  I'm actually a fan of the BSD license in this context.  It means commercial code can easily statically compile with Libdrizzle which in-turn means more MySQL Server/Percona Server/MariaDB Server adoption and more support contracts for Oracle/Percona/SkySQL/etc...

Even if no one uses the code I still find it a fun challenge and something I will continue for as long as I can.

The Future of Libdrizzle

In the next few days we will be releasing Libdrizzle 5.1.3, this will contain all the work that was done in Seattle (and my flights to/from Seattle) including a few nice API improvements.  Along with this I will create the first release of Drizzle Tools.

Drizzle Tools is a project which contains utilities for use with a MySQL server, at the moment the only one included is drizzle_binlogs which connects to a remote MySQL server as a slave and downloads the binary logs to local files as a back.  It also has a continuous mode which effectively gives an up-to-the-second backup of the data.  I have the code half-written for the next tool which will not make the first release but should be useful to many people.

Of course anyone is welcome to come hack on Librizzle and Drizzle Tools, file bugs, ask questions, etc...  The more knowledge share the better :)

There will be more posts soon showing how to use Libdrizzle and drizzle_tools as well as announcing the new features, so watch this space!