SkySQL - The Return of the Jedi

The last few weeks have been particularly quiet from me on the blogging front.  Behind the scenes things have been quite the opposite so here is a summary of things past, present and future.

Rackspace and Drizzle

If you have read my last 'Last Week in Drizzle' post you will know that Rackspace are no longer supporting Drizzle.  They have done a fantastic job so far and have decided to pass the baton to other companies.  As for the staff, they wished to redeploy us to other teams which is something I personally was not keen on.  I would rather remain within the MySQL/Drizzle sphere which I would have no longer been able to do effectively inside Rackspace any more.

Drizzle itself will go on to do great things without Rackspace, there are a number of companies that announced support for Drizzle during the O'Reilly MySQL Conference and Expo and Google Summer of Code is still going ahead as planned.

MySQL Conference

For me personally it was the busiest conference I have ever attended, this is mostly down to the three talks I had to give on top of booth duty, meetings and Drizzle Developer Day.  I had some fantastic feedback from people whilst there on many subjects such as Drizzle and the MySQL 5.1 Plugins Development book.  It was great to meet up with old friends and make some new ones and I hope that the conference will continue for many years to come.

SkySQL

The day after returning from the conference I started my new role as Senior Sustaining Engineer at SkySQL (very jetlagged and in hindsight I should have given myself a day or two to recover!).  In this role I not only go back into supporting customers but also developing tools around the MySQL/Drizzle sphere.  I feel very honoured to be working with the team (many of whom I am working with for a second time), they have really done a great job of capturing the traditional MySQL spirit.

One of the first things I have been working on is a new version of mydumper, once this is ready I will create a separate blog post about it.  I think it is a fantastic tool and hope that it will be able to help many users in the future.

Google Summer of Code

SkySQL have encouraged me to continue my work on Drizzle which I have also been doing.  As part of this I am a mentor for Google Summer of Code, a student called Olaf van der Spek will be working on improving the libdrizzle client API under my guidance.  Something I am very much looking forward to.

The Return of the Jedi

So, I am back in a support type role whilst also developing useful tools and patches to enhance the usability of MySQL, I will also be blogging more and getting involved in the community/ecosystem in other ways.  This is very similar to what I was doing at Sun/Oracle but for a company designed from the ground up to be much better for the staff and customers.  I am looking forward the the bright future of SkySQL.

Last Week in Drizzle

Welcome to this week's edition of Last Week in Drizzle.  Unfortunately I could not write this at the Drizzle Developer Day because it is was much busier than I expected.  So this one had to wait until I landed in the UK :)

O'Reilly MySQL Conference and Expo

Last week was the MySQL Conference which at this point should probably be called the MySQL & friends or the Open Database Conference.  We had many talks, great exposure and some fantastic questions and feedback of ideas we had never thought of during the week.  I urge anyone who wasn't there to watch Brian Aker's keynote on the State of Drizzle.

Drizzle Developer Day

On the Friday after the conference we had Drizzle Developer Day which contained people from every level, new users to some of the biggest names in MySQL development.  A great many topics were discussed such as the catalogs work, replication and storage engines.  There were also discussions governance and I hope there will be announcements on this in the near future.

Development Goes On!

All last week the Drizzle Developers at Rackspace were giving talks and meeting people at the conference so we didn't get a lot of time for actually writing code, but there have been many branches and merge requests thanks to the fantastic community around Drizzle.  I still haven't caught up with all the work merged in within the last week!

Drizzle Support

A few companies have come out in the last week offering support services for Drizzle such as Blue Gecko and Percona which is fantastic to see.  This is on top of companies such as SkySQL already providing support for Drizzle.  There could also be announcements from more companies in the near future, so watch this space!

Rackspace and Drizzle

If you have watched the keynote from Brian Aker referenced above you will notice near the end he talks briefly about the developers getting hired by other companies.  Rackspace have done a fantastic job in supporting and funding us right up through the GA but unfortunately can no longer go on providing resources for us.  What does that mean for us?  Most of the developers who work for Rackspace are moving to other companies, many of whom will continue to work on Drizzle.  What does the mean for Drizzle?  Not a whole lot really, development will continue as before.  The great thing about Drizzle is the Rackspace developers were actually the minority, there are many other companies as well as community developers involved.  The features that were originally planned will continue to be developed, Google Summer of Code will also go on as before.

My Next Adventure

My last day at Rackspace was last Friday, today I start my new adventure as Senior Sustaining Engineer at SkySQL.  This means I will still get to work on Drizzle as well as providing fantastic support and development resources for MySQL and MariaDB.

Final Thoughts

I'd personally like to thank everyone for their fantastic feedback at the conference and developer day last week.  It is great to hear that people think we are on the right path with the technology and many of the ideas and discussions that came up last week will help shape the future of Drizzle.

As always if you have any feedback or topics you would like me to cover, please let me know.

Drizzle - The Icing on the Cake - part 1

As I'm sure all of you know already, today marks the GA release of Drizzle7.  But what was the recipe behind Drizzle?

1. Take the raw ingredients from previous delicious, well-tried recipes


2. Sieve out the lumps, separate the eggs and mix


3. Bake using many cooks in many ovens for around 3 years


4. Drizzle the special source on top

What is Drizzle?

There are many marketing buzzwords which can describe what Drizzle is such as "A lightweight, microkernel fork of MySQL optimized for the web and cloud".  To me such things are pretty meaningless.  So, lets start at the beginning...

Several people inside MySQL saw that the code could really do with re-factoring.  At the same time they believed that the focus was heading away from it's core web based installations.  They also loved open source, and whilst MySQL is open source, community contributions can be difficult.  These people got together inside Sun Microsystems (and other companies) to create a completely open development of a fork of MySQL 6.0 called "Drizzle".  They aimed to have it easy for new developers to pick up and develop on, moved many parts out to plugins thereby making it light on resources when features are not needed.

In 2010 the original development team moved to Rackspace and several more members were hired (including me), with the aim of Drizzle being used in it's cloud based products.  Even today the amount of active community contributors is higher than the amount of developers inside Rackspace working on Drizzle.

Differences From MySQL

I have been asked many times what the differences are between MySQL and Drizzle.  This is something I could probably write a book on now.  Something that should be clear at this stage is Drizzle is not MySQL, it was MySQL over 3 years but a lot has changed since then.  Having said that, applications that use MySQL can usually be converted to use Drizzle relatively easily.  For new users to Drizzle, here are a few of the key differences:

• Strictness - Drizzle doesn't assume what you mean (which can cause incorrectly recorded data).  For example trying to store an invalid ENUM will error instead of storing an empty value.


• Data Types - Drizzle has removed, altered and added data types to simplify things and become closer to the SQL standard.  For example:
  
  
  
  • There is no TINY/SMALL/MEDIUM INT, just INT (and BIGINT).
  
  
  • There is no TINY/MEDIUM/LONG TEXT/BLOB, there is just TEXT/BLOB.
  
  
  • TIMESTAMP supports microsecond precision.
  
  
  • UUID and a true BOOLEAN type added.
  
  
  
  


• Replication - Drizzle's replication uses Google Protocol Buffer messages so a replication reader can be written in any language in minutes.  The replication data is stored in InnoDB as part of the transaction as it is being committed so that writing the replication log is very fast.


• Development - Drizzle is developed using a completely open development model which I will discuss in part 2.


• Licensing - The main drizzle source is GPLv2 licensed, libdrizzle is BSD licensed and the docs (which are also included in the docs directory of the source) are CC SA 3.0 licensed.  There is no proprietary licensing for any part of Drizzle.

Compatibility With MySQL

Despite many changes there is still a great deal of compatibility with MySQL.  Drizzle speaks the MySQL protocol, so existing MySQL connectors for PHP/Perl/etc... will also connect to and query Drizzle.  The SQL syntax is still very similar to MySQL and on top of all this, drizzledump (which is very similar to mysqldump) can convert table structures and data from MySQL to Drizzle on-the-fly.

Drizzle also includes libdrizzle.  This is a BSD licensed client library written in C which can talk to MySQL and Drizzle servers, from our testing as well as the testing of developers who are integrating libdrizzle into their products it appears that libdrizzle performs better than libmysqlclient too.  Connectors for libdrizzle have been written for most widely used languages such as Python, Java, Perl and PHP.

Plugins

Drizzle uses a completely new plugin architecture so that almost everything is a plugin.  From storage engines, to functions, to protocols, to authentication and even query cache.  This makes it much simpler to switch off the parts you don't use as well as customising Drizzle for your unique application.  In total there are around 80 plugins bundled in the Drizzle source and several others available around the web.

Despite this we have tried to make this easy for most people by having the plugins that most people will use compiled and running by default.

Summary

It is almost impossible to get a feel for what Drizzle is like without trying it for yourself.  We have had some great feedback both positive and negative, and have made changes thanks to this feedback.  We are all very approachable on #drizzle on Freenode and the Drizzle mailing list.

In part 2 I will discuss the open development model behind Drizzle

Clearing the global status

If you want to clear local session status variables this is easy, simply execute FLUSH STATUS.  Unfortunately doing the same for global status variables is not as easy.

That is until now.  In the Drizzle7 beta release 2010-11-08 is a new command: FLUSH GLOBAL STATUS

I created this command to fulfil a Rackspace feature request but it appears this has also been a MySQL feature request in the past.  The implementation would be very different in MySQL since the status variables and sessions work differently, but it would certainly be good to see in a future release, and it appears patches already exist for it.

Rackspace Rookie Orientation

I have been meaning to write about Rackspace's Rookie Orientation for a while, especially since Stewart Smith already wrote about his experience.  My orientation happened in the London offices in early October, it is a 4 day session in which we learn about and indeed get a feel for the culture at Rackspace as well as meeting many people and finding out about their roles along the way.  Yes there were presentations, but they were all done in a fun way.

Before Rackspace I have been to two orientations:

The first back in 2001 (during 9/11) was for a very large UK company and it was a week of team building and training us immoral sales tactics and even some things which would break trading standards laws (needless to stay I didn't hang around in that company for long).

The second was after one company I worked for got bought out by another (not Sun-Oracle), all I can really remember from this was the CEO used to get stoned a lot and got jailed for a publication a long time ago.

So what can I take away from the Rackspace Orientation?

I met some great people, we had a good time, we even made a video (which I won't show).  I got a real feel for the Rackspace culture which is very much what I hoped for.  Yes there were presentations, but presentations with explosions, a chicken, top-hats, wrists tied up in rope and one guy dressed as a semi-naked superhero.  This is not your average orientation :)

During the week we were split into four teams and there were two prizes.  The first was for the team who got the most points, my team (Rack'n'Rollers, see photo) won that award.  There was also a prize called 'Natural Born Racker' which was voted for by everyone at the orientation.  I won that prize and would really like to thank everyone who voted for me :)

My team's photo, I'm the one in the green top.

I admit I spent a lot of the week thinking about the work I need to catch up on, but it was an 'awesome' week.

One month at Rackspace

My official start date at Rackspace was the 31st August, so I have now been here a full month working on the Drizzle project, and so far it has been a really wild ride which I hope will continue for a long time to come.

First I should mention why I left Oracle/MySQL to kill all speculation on the subject.  I love the people I worked during my time at MySQL, they are a great and very knowledgeable team and still talk to some of them now.  Will Oracle be a good home for MySQL?  Its not for me to speculate and I will not comment on this.  My primary reason for leaving was administrative, I had a lot of payroll/expense/HR problems at Oracle after the Sun acquisition which led to a great deal of stress for my whole family.  I have always been a fan of Drizzle and have been contributing when I had spare time, I've also been a big fan of Rackspace and their work attitude/policies so I seized the opportunity when it became available.

During my time as a paid developer for the Drizzle project I have fixed roughly 30-35 bugs (its hard to get a real figure from Launchpad without opening every bug I have ever fixed).  I have also implemented the MySQL->Drizzle migration blueprint and made a start on some other cool new stuff which I will blog about soon.  We also have a beta release (if you have been living in a hole and haven't seen all the blog posts on this)!  There are two things I have found I love working on in Drizzle, diagnosing strange problems and improving the user experience.  I think the former has rubbed off from working in MySQL Support and the latter from working on the various websites at Dennis Publishing.

One of my worries was what my new team would be like, thankfully everyone in Drizzle is as warm and welcoming as my previous MySQL team and I think they are all great and very talented guys.  I'd like to thank everyone at Drizzle and Rackspace for giving me this opportunity so far and I hope I can contribute as much as possible to bring every success to both.

LinuxJedi, the new era

I have recently moved on from my roll as a Senior MySQL Support Engineer at Oracle to a Drizzle Developer at Rackspace.  As such my previous blog is now sitting idle so this is the replacement.

This new blog may have much less MySQL Cluster content, but I do hope to continue the same quality (I shall leave you to judge on how high that is!) of blog posts.