Cloud Users: Don't put your eggs in one basket

In my previous blog post I briefly mentioned that a premature cloud account removal meant that the Drizzle project lost a lot of data, including backups.  I'll go into a bit more detail here so cloud users can learn from our mistakes.  I will try to avoid using cloud puns here as much as possible :)

As with everything in life clouds fail.  No cloud I have ever seen so far can claim they have had 100% uptime since the beginning of hitting GA.  However much projects like Openstack simplify things for the operators of clouds they are complex architectures and things can fail.

But there is one element of human failure I've seen several times from multiple cloud vendors which is the problem that crippled Drizzle.  That is account deletion.

Most of Drizzle's websites and testing framework were running from a cloud account using compute resources.  Backups were made and automatically uploaded to the cloud file storage on the same account for archiving.  This was our mistake (and we have definitely learnt from it).  We knew that at some point in the future the cloud accounts used would be migrated to a different cloud and the current cloud account terminated.  Unfortunately the cloud account used was terminated prematurely.  This meant that all compute instances and file storage was instantly flushed down the toilet.  All our sites and backups were instantly destroyed.

This is not the only time I have seen this happen.  There have been two other instances I know of in the last year where an accidental deletion of a cloud account has meant that all data including backups were destroyed.  Luckily in both those cases the damage was relatively minor.  I actually also lost a web server due to this problem around the same time as Drizzle was hit.

The Openstack CI team do something quite clever but relatively simple to mitigate against these problems and continue running.  They use multiple cloud vendors (last I checked it was HP Cloud and Rackspace).  When your commit is being tested in Jenkins it goes to a cloud compute instance in whatever cloud is available at the time.  So if a vendor goes down for any reason the CI can still continue.

I highly recommend a few things to any users of the cloud.  You should:

1. Make regular offsite backups (and verify them)
2. If uptime is important, use multiple cloud providers
3. Use Salt, Ansible or similar technology so that you can quickly spin your cloud instances up again to your requirements at a moments notice

Patrick Galbraith, a Principal Engineer who works with me at HP's Advanced Technology Group is currently working on a way to enhance libcloud to work with HP Cloud better so that it is easy to seamlessly use multiple clouds.  We are also working on several enhancements to salt and ansible.  Both very promising technologies when it comes to cloud automation.

The way I see it, no one should be putting all their cloud eggs in one basket.

Is Drizzle dead?

Yesterday someone opened a Launchpad question asking "is Drizzle dead?".  I have answered that question on Launchpad but wanted to blog about it to give a bit of background context.

As I am sure most of the people who read this know, Drizzle is an Open Source database which was originally forked from the alpha version of MySQL 6.0.  At the time it was an extremely radical approach to Open Source development, many features were stripped out and re-written as plugins to turn it into a micro-kernel style architecture.  Every merge request was automatically throughly tested on several platforms for regressions, memory leaks and even positive/negative performance changes.

In fact Drizzle has influenced many Open Source projects today.  Openstack's Continuous Integration was born from the advanced testing we did on Drizzle.  MariaDB's Java connector was originally based on Drizzle's Java connector.  Even MySQL itself picked up a few things from it.

Development of Drizzle started off as a "What if?" R&D project at Sun Microsystem spearheaded by Brian Aker.  Once Oracle acquired Sun Microsystem a new corporate sponsor was found for Drizzle, Rackspace.

Rackspace hired all the core developers (and that is the point where I joined) and development progressed through to the first GA release of Drizzle.  Unfortunately Rackspace decided to no longer sponsor the development of Drizzle and we had to disband.  I've heard many reasons for this decision, I don't want to reflect on it, I just want to thank Rackspace for that time.

Where are we now?  Of the core team whilst I was at Rackspace:

• Brian and I work for HP's Advanced Technology Group.
• David Shrewsbury, Monty Taylor and Patrick Crews all work for various part of HP Cloud.
• Stewart Smith works for IBM's Linux Technology Centre.

So, back to the core question: "Is Drizzle dead?".  The core team all work long hours in our respective jobs to make some awesome Open Source products and in what little spare time we have we all work on many Open Source projects.  Unfortunately splitting our time to work on Drizzle is hard, so the pace has dramatically slowed.  But it isn't dead.  We have been part of Google Summer of Code, we still get commits in from all over the place and Drizzle is still part of the SPI.

Having said this, Drizzle no longer has a corporate sponsor.  Whilst Drizzle can live and go on without one, it is unlikely to thrive without one.

Another thing that is frequently asked is: "What happened to the docs and wiki?".  Drizzle being a cloud databases had all of its development and public documentation servers hosted in the cloud.  Unfortunately the kill switch was accidentally hit prematurely on the cloud account used.  This means we not only lost the servers but the storage space being used for backups.  This also affected other Open Source projects such as Gearman.  The old wiki is dead, we cannot recover that content.  The docs were auto-generated from the reStructuredText documentation in the source.  It was just automatically compiled and rendered for easy reading.

What I would personally like to see is the docs going to Read The Docs automatically (there is an attempt to do this, but it is currently failing to build) and the main site moved to DokuWiki similar to the new Gearman site.

As for Drizzle itself...  It was in my opinion pretty much exactly what an Open Source project should be and indeed was developing into what I think an Open Source database should be.  It just needs a little sponsorship and a core team that are paid to develop it and mentor others who wish to contribute.  Given that it was designed from the ground-up to be a multi-tenant in-cloud database (perfect for a DBaaS) I suspect that could still happen, especially now projects like Docker are emerging for it to sit on.

Mydumper 0.2.3 released!

Today marks the release of mydumper 0.2.3.  Mydumper is a multi-threaded high-performance data dumper (and loader) for MySQL and Drizzle written in C. This is a minor bugfix release whilst I work on the upcoming 0.5 version. The end goal here will be that there is always a 'stable' and 'development' version. 0.2 will be the first stable version and will only have bug fixes. 0.5 will be the first 'development' version where the next planned set of new features will hit.

So, the changes since 0.2.2 are:

• Drizzle support now fully works again


• Fixes so mydumper compiles in FreeBSD (thanks to Kirill A. Korinskiy)

If you wish to try this version the source can be downloaded here.

Last Week in Drizzle

Welcome to the second edition of Last Week in Drizzle.  The diff of the trunk between last Friday and right now is just under 23,000 lines in size, so I will do my best to summarise the important parts of this.

Replication

A lot of great work has gone into solidifying replication this week.  The slave code has not been merged yet since it has been triggering bugs in the Solaris compiler (see further down this post) but we plan to have it included in the next few hours ready for RC2.  For a quick summary of our current progress I turn to Patrick Crews:

"In our testing, we create a master-slave setup, then run the transaction log tests that we have been using since we first started beating on the trx log.

There are a variety of different grammars that produce transactions (autocommit=off) and single queries (with autocommit=on).  The various grammars produce different levels of valid queries - some make more valid queries and produce more deadlocks in multi-threaded testing, others create invalid queries to test how the log handles bad input.  We test with simple scenarios like 100 cycles and 1 thread, then move on to more complex tests like 10000 cycles and 10 threads as well as setup 1 million cycle tests to stress the server long-term.

There are still bugs to be found, but we can say with confidence that basic replication, even in highly concurrent / high stress (lots of deadlocks, rollbacks, and good commits) scenarios is working well.  Data is replicated correctly and that is HUGE!

We'll now be moving onto other tests like different configurations (master + slave restarts and crashes, adding a slave to a populated master, KILLing queries, etc)."

More information about setting up replication can be found in Patrick's blog post and David Shrewsbury will have a blog post on it shortly.  Documentation on the docs site will be coming soon.

Race to GA

Our RC2 release is due for tagging on the 28th of February which will include the stabilised replication code.  So the release schedule as last week is:

RC1 - 14th February 2011 Released
RC2 - 28th February 2011
GA - 14th March 2011

BOOLEAN data type

There has been a slight change to the output of the new native BOOLEAN data type.  In the first RC release the MySQL client API returned this as the varchar 'TRUE' or 'FALSE'.  Unfortunately some languages such as PHP and Python did not handle this too well, so thanks to Monty Taylor we now call this TINYINT on the wire for MySQL protocol and return 0 and 1.  Our command line utilities know when they are connecting to Drizzle instead of MySQL and will do the conversion to 'TRUE' or 'FALSE' when displaying output.

Docs day

Wednesday 23rd was our documentation day and I would like to thank everyone who took the time to read through the docs and make suggestions and fixes.  Since Wednesday morning the diff of the docs directory in trunk is just under 4500 lines in size and there are plenty more changes on their way based on the feedback gained.  I'd also like to thank Marisa Plumb, our main documentation writer, for all her hard work so far.

BIT operators

Brian Aker has (by popular demand) added back the SQL bit operators that MySQL has into Drizzle, this includes bit shifting operators.

Message verboseness

Brian has also added a new option to drizzled to set how verbose the output messages should be, defaulting to ERROR only.  More information can be found in the mailing list.

Solaris Jenkins Slave

We have hit several bugs in the Solaris compiler in the last week, Monty Taylor has been fire-fighting this but for now our Jenkins Solaris slave is not creating working builds.  This does mean that there is a chance that RC2 won't be fully tested on Solaris when released.

PHP module

I have been working with the PECL guys to get the Drizzle module working again.  The current release will not compile with the libdrizzle inside Drizzle.  The fix for this has been pushed to the SVN trunk in PECL and we are looking to generate a release of this soon.  In the mean time, the mysql and mysqli PHP connectors work great with Drizzle.

I have also received a lot of feedback from the PHP community that many of them use PDO for database connections.  We currently have no PDO module so I am working on writing one which hopefully will be ready for testing in the next week.

Final Thoughts

A massive amount of bug fixes and improvements have gone into the trunk ready for RC2, including many code clean-ups from Olaf van der Spek who is a new contributor to Drizzle.  I for one am looking forward to a very exciting GA release.

As always if you have any feedback or topics you would like me to cover, please let me know.

Last Week in Drizzle

It has been a while since we have done one of these so I thought I might try and resurrect the tradition.  So here is my first "Last Week in Drizzle".

Replication

The original plan for replication was to use a Tungsten Replicator based solution to transfer the transaction logs (similar to MySQL's binary logs).  Unfortunately this can't be completed in time for the GA release so we have switched to a master-slave solution similar to MySQL.  The Tungsten solution is still something we plan to finish though.

Replication events are stored using Google Protocol Buffer messages in an InnoDB table, these events are read by the slave, stored locally and applied.  The advantage of the Google Protocol Buffer messages is a script or program can be knocked up in pretty much any language in minutes to read our replication log.

Unfortunately this sudden change in replication method means we could not complete the slave code in time for the RC release, which in turn means we are creating a second RC release as explained further down in this blog post.

Special thanks goes to David Shrewsbury, Patrick Crews and Joe Daly for making this happen.

New Release

Our first RC has been released this week.  In this release we have:

• Drizzle server can now fork to background via. --daemon.  This was primarily implemented to help RedHat/Fedora init.d scripts.


• Implicit Cartesian Joins no longer work this is to prevent runaway queries.


• Improvements to the replication transaction log.


• Many other bug fixes and improvements.

Race to GA

Due to the late entry replication code we intend to have one more RC whist we test it to death in as many horrid ways as Patick Crews can find.  So the current release plan for Drizzle7 is now:

RC2 - 28th February 2011
GA - 14th March 2011

New RPM Repository

Derks has created a new RPM repository for us at rpm.drizzle.org, more details on this can be seen here.

Windows Jenkins Slave

Monty Taylor has created a Windows slave for our Jenkins Continuous Integration testing system.  This means we now test every trunk merge for libdrizzle regressions in Windows.

Docs Day

On Monday 21st February we have our docs day, the developers will be reviewing the entire docs site for technical errors and any improvements that can be made (such as missing topics).  We encourage anyone who would like to improve the quality of our docs to join in this effort, contact us on #drizzle on Freenode or file a bug if you spot anything we could improve on.

Update 2011-02-20: We are postponing this until Wednesday 23rd February due to documentation merges which won't have quite hit trunk by Monday.

Final Thoughts

I'm going to try and do one of these every week, so if you have any feedback or topics you would like me to cover, please let me know.