Last Week in Drizzle

Welcome to this week's edition of Last Week in Drizzle.  Unfortunately I could not write this at the Drizzle Developer Day because it is was much busier than I expected.  So this one had to wait until I landed in the UK :)

O'Reilly MySQL Conference and Expo

Last week was the MySQL Conference which at this point should probably be called the MySQL & friends or the Open Database Conference.  We had many talks, great exposure and some fantastic questions and feedback of ideas we had never thought of during the week.  I urge anyone who wasn't there to watch Brian Aker's keynote on the State of Drizzle.

Drizzle Developer Day

On the Friday after the conference we had Drizzle Developer Day which contained people from every level, new users to some of the biggest names in MySQL development.  A great many topics were discussed such as the catalogs work, replication and storage engines.  There were also discussions governance and I hope there will be announcements on this in the near future.

Development Goes On!

All last week the Drizzle Developers at Rackspace were giving talks and meeting people at the conference so we didn't get a lot of time for actually writing code, but there have been many branches and merge requests thanks to the fantastic community around Drizzle.  I still haven't caught up with all the work merged in within the last week!

Drizzle Support

A few companies have come out in the last week offering support services for Drizzle such as Blue Gecko and Percona which is fantastic to see.  This is on top of companies such as SkySQL already providing support for Drizzle.  There could also be announcements from more companies in the near future, so watch this space!

Rackspace and Drizzle

If you have watched the keynote from Brian Aker referenced above you will notice near the end he talks briefly about the developers getting hired by other companies.  Rackspace have done a fantastic job in supporting and funding us right up through the GA but unfortunately can no longer go on providing resources for us.  What does that mean for us?  Most of the developers who work for Rackspace are moving to other companies, many of whom will continue to work on Drizzle.  What does the mean for Drizzle?  Not a whole lot really, development will continue as before.  The great thing about Drizzle is the Rackspace developers were actually the minority, there are many other companies as well as community developers involved.  The features that were originally planned will continue to be developed, Google Summer of Code will also go on as before.

My Next Adventure

My last day at Rackspace was last Friday, today I start my new adventure as Senior Sustaining Engineer at SkySQL.  This means I will still get to work on Drizzle as well as providing fantastic support and development resources for MySQL and MariaDB.

Final Thoughts

I'd personally like to thank everyone for their fantastic feedback at the conference and developer day last week.  It is great to hear that people think we are on the right path with the technology and many of the ideas and discussions that came up last week will help shape the future of Drizzle.

As always if you have any feedback or topics you would like me to cover, please let me know.

Last Week in Drizzle

Welcome to this week’s Last Week in Drizzle.  This again will be a relatively short edition as the 2011 O'Reilly MySQL Conference and Expo is next week and I'm currently packing for it!

Drizzle in Real Time Data Visualization

Many of you will have seen the awesome real time data map of Mozilla's downloads on their glow site.  One thing that got me really excited this week was work by Marcus Eriksson to do the same thing using Drizzle and it's RabbitMQ connector.  The live demo of this has been hosted on a Rackspace cloud server and can be found here.

Percona's Contributions

It has been very encouraging this week to see staff at Percona submit merge requests.  So far there have been several InnoDB branches committed which add cool features such as saving the buffer pool to disk (actually identifiers for the pool so it is really small) for restoring upon server start as well as performance improvements.  Many thanks to the Percona guys for you hard work so far!

Drizzle Amazon AMI

Bluegecko have created a 32bit Drizzle AMI image using RPMs (which I believe was not an easy feat given how different it is to other OS's we compile for).  Anyone wanting to try this should check out their blog post on it which gives details on how to access the AMI.

Google Summer of Code

We have had many submissions to Google Summer of Code this year and it is unfortunate that we cannot take everyone on via. GSoC.  But there has been some awesome submissions this year, we are really impressed with the amount of effort people have put in their proposals.  If you want to get a last-minute submission in you can do so via. the GSoC site, the deadline is 12:00 Pacific Time today.

Final Thoughts

That about wraps up the big events of the past week.  If you are in Santa Clara, CA next week please come along to one of the many Drizzle talks at the 2011 O'Reilly MySQL Conference and Expo we will also have a booth in the Expo hall where you can come and chat to us.

Next week's edition will be coming to you live from the Drizzle Developer Day, Friday 15th April at the Hilton across the road from the Hyatt (where the conference is held).  As always if you have any feedback or topics you would like me to cover, please let me know.

Last Week in Drizzle

Welcome to this week's Last Week in Drizzle.  Today will be a relatively short edition due to the work everyone is doing preparing for the 2011 O'Reilly MySQL Conference and Expo and Google Summer of Code.

First Fremont Tarball

The first tarball of the Fremont development branch of Drizzle was created this week, following our tradition of releasing a tarball every two weeks.  It includes many experimental things such as the libdrizzle-2.0 separation and the multiple master to single slave replication.

For those wanting the stable release we suggest sticking to the Elliott branch which our GA was cut from.  New releases for this will be created much less frequently and will only include bug fixes.

Xtrabackup

Stewart Smith's work on integrating Xtrabackup into Drizzle has now been merged into Fremont.  When installing it will create a binary called 'drizzlebackup.innobase'.

Multi-Master Replication

By this I mean the work going into replication of multiple masters to a single slave.  Patrick Crews has written the first part of blog post covering the testing of this new feature.

Drizzle Migration

I have written an article for The H Online called 'A migrator's guide to Drizzle' which was published yesterday.  For anyone migrating from MySQL to Drizzle this should act as a useful guide.  I will also be giving a talk at the 2011 O'Reilly MySQL Conference and Expo entitled 'MySQL to Drizzle, stress free migration' which will be along similar lines.

FreeBSD

FreeBSD support has initially been dropped for Drizzle due to issues we were having with that platform.  It may well be coming back soon with the aid of Greg Larkin from FreeBSD who has offered to help get things running again.

Final Thoughts

If you happen to be in the vicinity of Santa Clara, CA between the 11th and 14th April we have quite a few talks on Drizzle as well as a booth in the expo hall.  It is a great opportunity to meet the team behind Drizzle.

As always if you have any feedback or topics you would like me to cover, please let me know.

Last Week in Drizzle

Welcome to this week's (slightly late) edition of Last Week in Drizzle.  This week sees the kick-off of many new features for the next release of Drizzle codenamed 'Fremont' and the mailing list is a hive of activity around Google Summer of Code.  I apologise for publishing a few days late this week and will try and stay on-track for future editions.

Fremont

In the tradition of Drizzle using Seattle road names in alphabetical order for codenames the next release of Drizzle is codenamed 'Fremont' (the current GA release is codenamed 'Elliott').  Monty Taylor has outlined the merge process going forward as can be seen in his mailing list post.

Google Summer of Code

We have been accepted for Google Summer of Code 2011 and are getting a lot of interest from potential applicants.  If you are interested in working on the Drizzle project as part of GSoC we have the following recommended instructions:

1. Check out our wiki page on potential projects


2. Email the mailing list with an introduction about yourself and join the Freenode #drizzle IRC channel to chat to us


3. Look at our low-hanging-fruit tasks and try to take one or two on.  This gets you used to the code and launchpad processes as well as gives us an insight into your abilities

Xtrabackup

Stewart Smith has been working hard on integrating Percona's Xtrabackup with Drizzle.  Xtrabackup is an online backup tool for InnoDB much like MySQL's Enterprise Backup.  This is nearly ready for merging into Drizzle and Stewart has written a great blog post on the subject on his blog.

Catalogs

Stewart has been a real busy guy this last week, another project he has been working on is getting catalogs support working with more than one catalog.  For those not familiar with catalogs they are a way of totally isolating one user's databases from another, similar to having multiple installations of Drizzle in one box but all running from one daemon.  In the GA release a lot of the framework already existed for catalogs and everything in it runs from a catalog called 'local'.  More information on the progress Stewart has made can be found on his blog post.

Libdrizzle 2.0

In Fremont we are working towards Libdrizzle 2.0.  This is a C++ version of Libdrizzle with a C compatible API.  Eventually it will contain new features such as native sharding (we are still working on filling out a potential features list for it).  For now in the Drizzle trunk you can see libdrizzle has been moved to libdrizzle-1.0 and a new libdrizzle-2.0 directory exists for the new work.

Multi-Master Replication

David Shrewsbury has been working on multi-master replication in Drizzle with a beta release ready to try.  By multi-master I mean having multiple masters write to a single slave.  For more information on this work take a look at his blog post.

Final Thoughts

Development is starting to move forward at a rapid pace for our next GA and we have had a lot of branches merged that I haven't discussed here from people such as Olaf van der Spek who has contributed a lot towards code cleanups.

As always if you have any feedback or topics you would like me to cover, please let me know.

Last Week in Drizzle

Welcome to the latest edition of Last Week in Drizzle.  This week we announced our GA release!!  Interest in Drizzle in the last week has been much higher than anticipated, this blog alone got 4,500 visitors on Wednesday! (which was also a nice test for the Drizzle database powering it)

GA Release

So, on Tuesday the tarball was cut for our GA release called Drizzle7.  Most of the changes from the last week relate to code cleanup, documentation and test suite improvements so that we could keep the codebase stable ready for the release (also many of us are busy writing conference talks around now :) ).  For a quick summary of what to expect in Drizzle7 and the future you can see my three-part special called "Drizzle - The Icing on the Cake": part 1 part 2 part 3

There have also been blog posts from several members of the Drizzle team: Brian Aker, Patrick Crews, Stewart Smith and more can be found on Planet Drizzle

Drizzle in the Media

We have had huge coverage by many technology publications which has been fantastic to read, I'll try to link to some of them here but you should be able to find more by searching Google News:

• The H: Cloud and web database Drizzle reaches general availability


• The Register: Drizzle: Big-Data-happy MySQL fork debuts (some technical inaccuracies in this, HailDB isn't the default engine, InnoDB is, but still great to see The Register covering us)


• InternetNews.com: Open Source Drizzle database now GA - Should Oracle worry?


• InfoWorld: Non-Oracle MySQL fork deemed ready for prime time


• NetworkWorld: Is the 'database for the cloud' ready? Drizzle GA released


• ZDNet: MySQL fork Drizzle gets general release


• Rackspace Cloud Blog: Drizzle7: The First GA release of the Drizzle Project (OK, I wrote that one ;) )

There has also been a lot of coverage in foreign media which is also fantastic to see.

Drizzle Downloads

The only metric we have for the amount of downloads for Drizzle is the source downloads.  We can't record the usage of the Ubuntu PPA or the installations using the pre-release of Ubuntu Natty (which has Drizzle in it's repositories) and currently don't log the RPM yum repository.  But on source downloads alone there were more downloads in the first 2 days of GA than the 2 weeks of the RC2 release (I count a shade under 400 downloads at the time of publishing this).  On top of this many more people have come to the #drizzle IRC channel on Freenode to ask us questions and even one or two minor bugs have already been found by new users.

Criticisms

The biggest criticism I have seen so far is the name 'Drizzle' for a database.  I personally like the name 'Drizzle', but I come from the UK where Drizzle is a very regular weather condition.  I actually think that is a really good thing, if the only big complaint is the name we most be going right somewhere :)

Fremont

Development has already started on the next version of Drizzle codename Fremont (Drizzle7's codename was Elliott).  This time we aim to make you wait a little less time, with the next GA scheduled for later this year.  For those who haven't guessed it already the codenames for Drizzle are based on road names in Seattle (another place where drizzle regularly happens).  It hasn't yet been decided whether this will be Drizzle7.1 or Drizzle8 and I'm sure we would take suggestions on this at the upcoming Drizzle Developer Day.

Final Thoughts

The overall feedback from the last week has been fantastic.  I'd like to thank everyone who has given us great coverage, everyone who has tried Drizzle and last but not least everyone who develops Drizzle whether it be at Rackspace, another company or just a community developer.

As always if you have any feedback or topics you would like me to cover, please let me know.

Last Week in Drizzle

Welcome to this week's edition of "Last Week in Drizzle".  As an introduction this week I would like to quote John David Duncan's recent Facebook post: "And what's in the weather forecast for next week? Drizzle.".  Yes, our first GA release is due next week, does that mean the development pace has slowed?  Heck no!  Over 150,000 lines of bzr diff in the trunk since last week and quite a few branches still in the merge queue going through our extensive regression testing system.

Google Summer of Code

We have once again applied to be part of the Google Summer of Code program.  We had some great students last year and some new faces interested in being students on projects for Drizzle have already started taking on some low-hanging-fruit tasks to get them used to our code and processes.  We will have a sign-up form up soon so that anyone interested in being part of the program which I will blog about when ready.  In the mean time you can read our wiki page about participation and if you have any suggestions for projects this year, please let us know.

Race to GA

We are just a few short days away now from the first Drizzle GA.  The release schedule for Drizzle7 is as follows:

RC1 - 14th February 2011 Released
RC2 - 28th February 2011 Released
GA - 14th March 2011

Engine Removal

By "engine removal" I don't mean the poor state of my car but the fact that we have removed some of the bundled storage engines from Drizzle this week.  This is because some needed maintaining, some didn't quite fit in with Drizzle and some just plain didn't compile any more.  This also helps us as developers support Drizzle by concentrating on the storage engines that are important to users.  Will the removed engines be gone forever?  If there is demand for them and they can be maintained, they will return.  The removed engines are archive, blackhole, filesystem_engine, blitzdb, csv and pbxt.

Node.js

Mariano Iglesias has created a node.js binding for libdrizzle.  In his words "the libdrizzle binding is outperforming node.js mysql bindings by a factor of at least 2 to 1" which is great to hear, especially since it can be used against a MySQL server.  An example of how to use it can be found here.

Libdrizzle Only Option

Monty Taylor as added the configure option --without-server to go along with "make libdrizzle" which will only compile libdrizzle from the drizzle trunk.  This should help anyone who only requires libdrizzle from source and doesn't want to have the dependencies required for the server to get it.

Authentication Defaults

Brian Aker has outlined changes to authentication such as the requirement of a username to connect to Drizzle and only listening on localhost by default.  Further details can be found on the mailing list where he is also asking for feedback on changes.

Final Thoughts

This time of year is incredibly busy for us, preparing for the GA release whilst getting ready to give lots of conference talks and other such things.  But despite this spirits are still high.  I for one am very proud of what has been achieved in Drizzle by the team at Rackspace and other companies and community members involved.  I hope new users coming to Drizzle find it as exciting as we do.

As always if you have any feedback or topics you would like me to cover, please let me know.

Last Week in Drizzle

Welcome to the third edition of Last Week in Drizzle.  The diff of the trunk between last Friday and right now is just over 230,000 lines in size, 10x the size of the previous week!  This includes many changes to the documentation, code clean-ups and Patrick Crews' continued work on our new DBQP test suite.

Replication

David Shrewsbury (I'm going to spell his name correctly this week ;)) and Patrick Crews have been working hard on making replication even more rock solid.  The slave plugin is in, working and is stable with everything we can throw at it.

Drizzle developer day

We have a Drizzle Developer Day at the 2011 O'Reilly MySQL Conference and Expo.  Anyone is welcome to come and learn, contribute and make suggestions about Drizzle.  It will be on Friday 15th April 9:30 a.m. to 5:00 p.m. although the exact location is to be confirmed it will likely take place in the Santa Clara Convention Centre.

If you wish to come along please sign-up here.

Race to GA

Our RC2 was released on the 28th of February and included the stabilised replication code amongst other fixes.  So the release schedule for Drizzle7 is:

RC1 – 14th February 2011 Released
RC2 – 28th February 2011 Released
GA – 14th March 2011

SQLAlchemy

Monty Taylor has been working hard to get Drizzle working with SQLAlchemy which has a very rigorous test suite.  This will help make it easier for Drizzle to act as a data store for OpenStack Compute which uses SQLAlchemy.

Libdrizzle

We have noticed that people have been downloading libdrizzle from the old libdrizzle Launchpad page.  Back in October we merged libdrizzle into the main Drizzle trunk and since then all bug fixes/development has happened there.  We highly recommend not using the old libdrizzle (which has known bugs) and we are in the process of shutting down the old libdrizzle development page.

Drizzle module for PHP

We have started developing the Drizzle module for PHP in Launchpad and have created a new release of this which is compatible with the libdrizzle in the Drizzle trunk.  We are working to get the fixes in PECL but in the mean time we will be developing in Launchpad and basing any binary releases from this.

Of course, Drizzle is compatible with the MySQL protocol so the existing MySQL functions and classes will work with Drizzle.

Drizzle module for Perl

Patrick Galbraith has released version 0.303 of DBD::Drizzle which contains several fixes for talking to a Drizzle server.

Final Thoughts

We are pretty much at the homestretch for the first GA, years of work from a huge amount of contributors will have a stable release.  Thanks to the many people who have helped make this happen.  We receive valuable feedback every day and all of it goes to make Drizzle a better product.

As always if you have any feedback or topics you would like me to cover, please let me know.

Last Week in Drizzle

Welcome to the second edition of Last Week in Drizzle.  The diff of the trunk between last Friday and right now is just under 23,000 lines in size, so I will do my best to summarise the important parts of this.

Replication

A lot of great work has gone into solidifying replication this week.  The slave code has not been merged yet since it has been triggering bugs in the Solaris compiler (see further down this post) but we plan to have it included in the next few hours ready for RC2.  For a quick summary of our current progress I turn to Patrick Crews:

"In our testing, we create a master-slave setup, then run the transaction log tests that we have been using since we first started beating on the trx log.

There are a variety of different grammars that produce transactions (autocommit=off) and single queries (with autocommit=on).  The various grammars produce different levels of valid queries - some make more valid queries and produce more deadlocks in multi-threaded testing, others create invalid queries to test how the log handles bad input.  We test with simple scenarios like 100 cycles and 1 thread, then move on to more complex tests like 10000 cycles and 10 threads as well as setup 1 million cycle tests to stress the server long-term.

There are still bugs to be found, but we can say with confidence that basic replication, even in highly concurrent / high stress (lots of deadlocks, rollbacks, and good commits) scenarios is working well.  Data is replicated correctly and that is HUGE!

We'll now be moving onto other tests like different configurations (master + slave restarts and crashes, adding a slave to a populated master, KILLing queries, etc)."

More information about setting up replication can be found in Patrick's blog post and David Shrewsbury will have a blog post on it shortly.  Documentation on the docs site will be coming soon.

Race to GA

Our RC2 release is due for tagging on the 28th of February which will include the stabilised replication code.  So the release schedule as last week is:

RC1 - 14th February 2011 Released
RC2 - 28th February 2011
GA - 14th March 2011

BOOLEAN data type

There has been a slight change to the output of the new native BOOLEAN data type.  In the first RC release the MySQL client API returned this as the varchar 'TRUE' or 'FALSE'.  Unfortunately some languages such as PHP and Python did not handle this too well, so thanks to Monty Taylor we now call this TINYINT on the wire for MySQL protocol and return 0 and 1.  Our command line utilities know when they are connecting to Drizzle instead of MySQL and will do the conversion to 'TRUE' or 'FALSE' when displaying output.

Docs day

Wednesday 23rd was our documentation day and I would like to thank everyone who took the time to read through the docs and make suggestions and fixes.  Since Wednesday morning the diff of the docs directory in trunk is just under 4500 lines in size and there are plenty more changes on their way based on the feedback gained.  I'd also like to thank Marisa Plumb, our main documentation writer, for all her hard work so far.

BIT operators

Brian Aker has (by popular demand) added back the SQL bit operators that MySQL has into Drizzle, this includes bit shifting operators.

Message verboseness

Brian has also added a new option to drizzled to set how verbose the output messages should be, defaulting to ERROR only.  More information can be found in the mailing list.

Solaris Jenkins Slave

We have hit several bugs in the Solaris compiler in the last week, Monty Taylor has been fire-fighting this but for now our Jenkins Solaris slave is not creating working builds.  This does mean that there is a chance that RC2 won't be fully tested on Solaris when released.

PHP module

I have been working with the PECL guys to get the Drizzle module working again.  The current release will not compile with the libdrizzle inside Drizzle.  The fix for this has been pushed to the SVN trunk in PECL and we are looking to generate a release of this soon.  In the mean time, the mysql and mysqli PHP connectors work great with Drizzle.

I have also received a lot of feedback from the PHP community that many of them use PDO for database connections.  We currently have no PDO module so I am working on writing one which hopefully will be ready for testing in the next week.

Final Thoughts

A massive amount of bug fixes and improvements have gone into the trunk ready for RC2, including many code clean-ups from Olaf van der Spek who is a new contributor to Drizzle.  I for one am looking forward to a very exciting GA release.

As always if you have any feedback or topics you would like me to cover, please let me know.

Last Week in Drizzle

It has been a while since we have done one of these so I thought I might try and resurrect the tradition.  So here is my first "Last Week in Drizzle".

Replication

The original plan for replication was to use a Tungsten Replicator based solution to transfer the transaction logs (similar to MySQL's binary logs).  Unfortunately this can't be completed in time for the GA release so we have switched to a master-slave solution similar to MySQL.  The Tungsten solution is still something we plan to finish though.

Replication events are stored using Google Protocol Buffer messages in an InnoDB table, these events are read by the slave, stored locally and applied.  The advantage of the Google Protocol Buffer messages is a script or program can be knocked up in pretty much any language in minutes to read our replication log.

Unfortunately this sudden change in replication method means we could not complete the slave code in time for the RC release, which in turn means we are creating a second RC release as explained further down in this blog post.

Special thanks goes to David Shrewsbury, Patrick Crews and Joe Daly for making this happen.

New Release

Our first RC has been released this week.  In this release we have:

• Drizzle server can now fork to background via. --daemon.  This was primarily implemented to help RedHat/Fedora init.d scripts.


• Implicit Cartesian Joins no longer work this is to prevent runaway queries.


• Improvements to the replication transaction log.


• Many other bug fixes and improvements.

Race to GA

Due to the late entry replication code we intend to have one more RC whist we test it to death in as many horrid ways as Patick Crews can find.  So the current release plan for Drizzle7 is now:

RC2 - 28th February 2011
GA - 14th March 2011

New RPM Repository

Derks has created a new RPM repository for us at rpm.drizzle.org, more details on this can be seen here.

Windows Jenkins Slave

Monty Taylor has created a Windows slave for our Jenkins Continuous Integration testing system.  This means we now test every trunk merge for libdrizzle regressions in Windows.

Docs Day

On Monday 21st February we have our docs day, the developers will be reviewing the entire docs site for technical errors and any improvements that can be made (such as missing topics).  We encourage anyone who would like to improve the quality of our docs to join in this effort, contact us on #drizzle on Freenode or file a bug if you spot anything we could improve on.

Update 2011-02-20: We are postponing this until Wednesday 23rd February due to documentation merges which won't have quite hit trunk by Monday.

Final Thoughts

I'm going to try and do one of these every week, so if you have any feedback or topics you would like me to cover, please let me know.