If you want to clear local session status variables this is easy, simply execute FLUSH STATUS. Unfortunately doing the same for global status variables is not as easy.
That is until now. In the Drizzle7 beta release 2010-11-08 is a new command: FLUSH GLOBAL STATUS
I created this command to fulfil a Rackspace feature request but it appears this has also been a MySQL feature request in the past. The implementation would be very different in MySQL since the status variables and sessions work differently, but it would certainly be good to see in a future release, and it appears patches already exist for it.
The /dev/null ramblings of a Linux Jedi
Sunday 21 November 2010
Friday 12 November 2010
Rackspace Rookie Orientation
I have been meaning to write about Rackspace's Rookie Orientation for a while, especially since Stewart Smith already wrote about his experience. My orientation happened in the London offices in early October, it is a 4 day session in which we learn about and indeed get a feel for the culture at Rackspace as well as meeting many people and finding out about their roles along the way. Yes there were presentations, but they were all done in a fun way.
Before Rackspace I have been to two orientations:
The first back in 2001 (during 9/11) was for a very large UK company and it was a week of team building and training us immoral sales tactics and even some things which would break trading standards laws (needless to stay I didn't hang around in that company for long).
The second was after one company I worked for got bought out by another (not Sun-Oracle), all I can really remember from this was the CEO used to get stoned a lot and got jailed for a publication a long time ago.
So what can I take away from the Rackspace Orientation?
I met some great people, we had a good time, we even made a video (which I won't show). I got a real feel for the Rackspace culture which is very much what I hoped for. Yes there were presentations, but presentations with explosions, a chicken, top-hats, wrists tied up in rope and one guy dressed as a semi-naked superhero. This is not your average orientation :)
During the week we were split into four teams and there were two prizes. The first was for the team who got the most points, my team (Rack'n'Rollers, see photo) won that award. There was also a prize called 'Natural Born Racker' which was voted for by everyone at the orientation. I won that prize and would really like to thank everyone who voted for me :)
I admit I spent a lot of the week thinking about the work I need to catch up on, but it was an 'awesome' week.
Before Rackspace I have been to two orientations:
The first back in 2001 (during 9/11) was for a very large UK company and it was a week of team building and training us immoral sales tactics and even some things which would break trading standards laws (needless to stay I didn't hang around in that company for long).
The second was after one company I worked for got bought out by another (not Sun-Oracle), all I can really remember from this was the CEO used to get stoned a lot and got jailed for a publication a long time ago.
So what can I take away from the Rackspace Orientation?
I met some great people, we had a good time, we even made a video (which I won't show). I got a real feel for the Rackspace culture which is very much what I hoped for. Yes there were presentations, but presentations with explosions, a chicken, top-hats, wrists tied up in rope and one guy dressed as a semi-naked superhero. This is not your average orientation :)
During the week we were split into four teams and there were two prizes. The first was for the team who got the most points, my team (Rack'n'Rollers, see photo) won that award. There was also a prize called 'Natural Born Racker' which was voted for by everyone at the orientation. I won that prize and would really like to thank everyone who voted for me :)
I admit I spent a lot of the week thinking about the work I need to catch up on, but it was an 'awesome' week.
Sunday 7 November 2010
New global buffer constraints in Drizzle
Both MySQL and Drizzle have buffers such as sort and join buffers which have size parameters to tune them. Unfortunately these settings are per-client, so if you have thousands of clients all using these buffers you could easily exhaust the RAM of a server. Especially if they are set quite high.
So in the latest Drizzle trunk (and the release due in the next few days) I have introduced global buffer constraints. These constraints make sure that between all the clients there is a cap on how much RAM can be allocated to a buffer.
This was a task the Rackspace Cloud team requested be implemented and I have also seen (and subsequently lost the link to) a blog which discusses the possibility of something similar in MySQL. If anyone has seen the blog post I am talking about please let me know :)
For each buffer type there is a new atomic counter which is keeping account of the allocations for each buffer type. If one of theses counters hits a cap then the query executing when the cap is hit will return an error. So far I have added this to 4 buffers which are sort buffer, join buffer, read buffer and read rnd buffer. The new options for these are:
By default all are set to 0 which means unlimited, and cannot be set lower than their associative settings (eg. sort-heap-threshold cannot be lower than sort-buffer-size).
I would typically expect this would be used most in cloud scenarios where the virtual servers could have a limited amount of RAM and swap available.
So in the latest Drizzle trunk (and the release due in the next few days) I have introduced global buffer constraints. These constraints make sure that between all the clients there is a cap on how much RAM can be allocated to a buffer.
This was a task the Rackspace Cloud team requested be implemented and I have also seen (and subsequently lost the link to) a blog which discusses the possibility of something similar in MySQL. If anyone has seen the blog post I am talking about please let me know :)
How this is implemented
For each buffer type there is a new atomic counter which is keeping account of the allocations for each buffer type. If one of theses counters hits a cap then the query executing when the cap is hit will return an error. So far I have added this to 4 buffers which are sort buffer, join buffer, read buffer and read rnd buffer. The new options for these are:
--join-heap-threshold
--read-buffer-threshold
--read-rnd-threshold
--sort-heap-threshold
By default all are set to 0 which means unlimited, and cannot be set lower than their associative settings (eg. sort-heap-threshold cannot be lower than sort-buffer-size).
I would typically expect this would be used most in cloud scenarios where the virtual servers could have a limited amount of RAM and swap available.
Tuesday 19 October 2010
IOUG's place in the MySQL Community
As I have previously stated it is quite rare that I blog about my feelings about things going on with Oracle in the MySQL community, others do this a lot better than me, but a blog post I've seen today has pushed me to write something about how I feel.
To me it implies that IOUG are trying to put themselves as the leaders of the MySQL community. I personally feel that they need to earn our trust first before trying this move.
I see that they mention the Collaborate conference, but no mention of the O'Reilly MySQL Conference & Expo (what I feel is the biggest social event on the MySQL calendar). Not only this but the Collaborate conference runs at the same time as the MySQL Conference. I don't feel this is a good start at earning trust.
EDIT: by the above I meant that promoting one but not the other as implied leaders of the community doesn't feel like a good start... Not that the timing was intentional. I should really blog *after* morning coffee :)
These are my personal feelings on the matter and they may not be shared by others, we are a community and are not going to all agree. But it would be good if IOUG could give us some confidence in this matter.
To me it implies that IOUG are trying to put themselves as the leaders of the MySQL community. I personally feel that they need to earn our trust first before trying this move.
I see that they mention the Collaborate conference, but no mention of the O'Reilly MySQL Conference & Expo (what I feel is the biggest social event on the MySQL calendar). Not only this but the Collaborate conference runs at the same time as the MySQL Conference. I don't feel this is a good start at earning trust.
EDIT: by the above I meant that promoting one but not the other as implied leaders of the community doesn't feel like a good start... Not that the timing was intentional. I should really blog *after* morning coffee :)
These are my personal feelings on the matter and they may not be shared by others, we are a community and are not going to all agree. But it would be good if IOUG could give us some confidence in this matter.
Friday 1 October 2010
Migrating from MySQL to Drizzle
One of Drizzle's great strengths is that it is not afraid to deviate from its MySQL origins. Unfortunately the side-effect of this is that it can make migration from MySQL to Drizzle more difficult. I have previous noted that drizzledump can do the schema migration for you. Now I will go into more details with some other information that may be useful if you are considering migration.
The first thing I should point out is that the default storage engine in Drizzle in InnoDB, MyISAM exists only for temporary tables and we have some other options such as PBXT and BlitzDB. As a kind-of side effect to this, there is no FULLTEXT indexes in Drizzle, so these have to be dropped as part of the migration.
When it comes to integer types we have two, INT and BIGINT. So TINYINT, SMALLINT, etc... need to be changed to INT. We also have no unsigned so an INT UNSIGNED needs to become BIGINT to prevent potential loss of data. Finally INT doesn't have column widths, for example the bit in braces in "INT(11)" so these can go.
There are also several changes to TIME/DATE storage. There is no TIME type in Drizzle, so Drizzledump converts this into an INT of seconds. MySQL accepts a date of 0000-00-00 whereas Drizzle's minimum date is 0001-01-01. So Drizzledump converts 0000-00-00 to NULL. Finally we have no YEAR data type in Drizzle. This should convert to INT using Drizzledump but as I am writing this I have found a bug here which I will fix shortly.
As far as TEXT/BLOB types go there is no tiny/medium/long. This is now just BLOB or TEXT.
We have one single character set in Drizzle, UTF8. So you should make sure data is exported in that character set. Different collations, however, are supported.
When you want to connect up your application to Drizzle, there is good news here. Drizzle supports the MySQL protocol and listens on port 3306! This means, for example, that your PHP app will connect straight to Drizzle thinking it is a MySQL server using the MySQL functions you are familiar with. There is no UNIX socket, so make sure your apps connect to 127.0.0.1 instead of localhost for local connections. There is no username/password by default but you can enable one of the authentication plugin modules if you require them.
The query syntax is pretty much the same as MySQL, there are things we don't have such as stored procedures and triggers. But for the most part this is unchanged.
The first thing I should point out is that the default storage engine in Drizzle in InnoDB, MyISAM exists only for temporary tables and we have some other options such as PBXT and BlitzDB. As a kind-of side effect to this, there is no FULLTEXT indexes in Drizzle, so these have to be dropped as part of the migration.
When it comes to integer types we have two, INT and BIGINT. So TINYINT, SMALLINT, etc... need to be changed to INT. We also have no unsigned so an INT UNSIGNED needs to become BIGINT to prevent potential loss of data. Finally INT doesn't have column widths, for example the bit in braces in "INT(11)" so these can go.
There are also several changes to TIME/DATE storage. There is no TIME type in Drizzle, so Drizzledump converts this into an INT of seconds. MySQL accepts a date of 0000-00-00 whereas Drizzle's minimum date is 0001-01-01. So Drizzledump converts 0000-00-00 to NULL. Finally we have no YEAR data type in Drizzle. This should convert to INT using Drizzledump but as I am writing this I have found a bug here which I will fix shortly.
As far as TEXT/BLOB types go there is no tiny/medium/long. This is now just BLOB or TEXT.
We have one single character set in Drizzle, UTF8. So you should make sure data is exported in that character set. Different collations, however, are supported.
When you want to connect up your application to Drizzle, there is good news here. Drizzle supports the MySQL protocol and listens on port 3306! This means, for example, that your PHP app will connect straight to Drizzle thinking it is a MySQL server using the MySQL functions you are familiar with. There is no UNIX socket, so make sure your apps connect to 127.0.0.1 instead of localhost for local connections. There is no username/password by default but you can enable one of the authentication plugin modules if you require them.
The query syntax is pretty much the same as MySQL, there are things we don't have such as stored procedures and triggers. But for the most part this is unchanged.
One month at Rackspace
My official start date at Rackspace was the 31st August, so I have now been here a full month working on the Drizzle project, and so far it has been a really wild ride which I hope will continue for a long time to come.
First I should mention why I left Oracle/MySQL to kill all speculation on the subject. I love the people I worked during my time at MySQL, they are a great and very knowledgeable team and still talk to some of them now. Will Oracle be a good home for MySQL? Its not for me to speculate and I will not comment on this. My primary reason for leaving was administrative, I had a lot of payroll/expense/HR problems at Oracle after the Sun acquisition which led to a great deal of stress for my whole family. I have always been a fan of Drizzle and have been contributing when I had spare time, I've also been a big fan of Rackspace and their work attitude/policies so I seized the opportunity when it became available.
During my time as a paid developer for the Drizzle project I have fixed roughly 30-35 bugs (its hard to get a real figure from Launchpad without opening every bug I have ever fixed). I have also implemented the MySQL->Drizzle migration blueprint and made a start on some other cool new stuff which I will blog about soon. We also have a beta release (if you have been living in a hole and haven't seen all the blog posts on this)! There are two things I have found I love working on in Drizzle, diagnosing strange problems and improving the user experience. I think the former has rubbed off from working in MySQL Support and the latter from working on the various websites at Dennis Publishing.
One of my worries was what my new team would be like, thankfully everyone in Drizzle is as warm and welcoming as my previous MySQL team and I think they are all great and very talented guys. I'd like to thank everyone at Drizzle and Rackspace for giving me this opportunity so far and I hope I can contribute as much as possible to bring every success to both.
First I should mention why I left Oracle/MySQL to kill all speculation on the subject. I love the people I worked during my time at MySQL, they are a great and very knowledgeable team and still talk to some of them now. Will Oracle be a good home for MySQL? Its not for me to speculate and I will not comment on this. My primary reason for leaving was administrative, I had a lot of payroll/expense/HR problems at Oracle after the Sun acquisition which led to a great deal of stress for my whole family. I have always been a fan of Drizzle and have been contributing when I had spare time, I've also been a big fan of Rackspace and their work attitude/policies so I seized the opportunity when it became available.
During my time as a paid developer for the Drizzle project I have fixed roughly 30-35 bugs (its hard to get a real figure from Launchpad without opening every bug I have ever fixed). I have also implemented the MySQL->Drizzle migration blueprint and made a start on some other cool new stuff which I will blog about soon. We also have a beta release (if you have been living in a hole and haven't seen all the blog posts on this)! There are two things I have found I love working on in Drizzle, diagnosing strange problems and improving the user experience. I think the former has rubbed off from working in MySQL Support and the latter from working on the various websites at Dennis Publishing.
One of my worries was what my new team would be like, thankfully everyone in Drizzle is as warm and welcoming as my previous MySQL team and I think they are all great and very talented guys. I'd like to thank everyone at Drizzle and Rackspace for giving me this opportunity so far and I hope I can contribute as much as possible to bring every success to both.
Wednesday 29 September 2010
Drizzle7 Beta Released! (now with MySQL migration!)
Today is a big day for us working on the Drizzle project. Our first official Beta! We have come a long way since the original MySQL 6.0 fork and we couldn't have done it without the many community contributors involved, so many thanks to everyone who has helped us to get here (you know who you are).
One feature I have been working on recently is a MySQL migration tool. This is basically a large modification to drizzledump which can do two extra things:
Firstly, it will automatically detect whether it is connecting to a MySQL or Drizzle server. When connecting to a MySQL server it will automatically convert the table structures and data to create a Drizzle compatible dump file.
On top of this there are new options so that drizzledump can connect straight to a Drizzle server at the same time as a MySQL server and pipe the data over, converting on-the-fly, without any intermediate dump file!
To help read up about how to use this, another new feature is the integration of Sphinx Documentation Generator. We are still in the process of writing documentation but we already have information on how to use the drizzledump migration options available.
The Drizzle server can already talk the MySQL protocol, so in many cases there will be little to no application modifications needed to switch from MySQL to Drizzle, you can even still use libmysqlclient!
Please test this and let us know what you think, we are always open to feedback.
One feature I have been working on recently is a MySQL migration tool. This is basically a large modification to drizzledump which can do two extra things:
Firstly, it will automatically detect whether it is connecting to a MySQL or Drizzle server. When connecting to a MySQL server it will automatically convert the table structures and data to create a Drizzle compatible dump file.
On top of this there are new options so that drizzledump can connect straight to a Drizzle server at the same time as a MySQL server and pipe the data over, converting on-the-fly, without any intermediate dump file!
To help read up about how to use this, another new feature is the integration of Sphinx Documentation Generator. We are still in the process of writing documentation but we already have information on how to use the drizzledump migration options available.
The Drizzle server can already talk the MySQL protocol, so in many cases there will be little to no application modifications needed to switch from MySQL to Drizzle, you can even still use libmysqlclient!
Please test this and let us know what you think, we are always open to feedback.
Tuesday 28 September 2010
Sphinx Documentation Generator RPMs
So, thanks to the excellent work of Monty Taylor we now use a documentation generator in Drizzle! He has added Sphinx Documentation Generator to the build system so that you can you can create documentation simply by doing:
(yes, you can create epubs!)
There are other formats available. Running 'make sphinx-help' will give you a full list of options available.
To use this you need to be using the latest trunk of Drizzle or the release that is due this week. A minimum python-sphinx 1.0 also needs to be installed (older versions will work, but some output will be incorrect). Unfortunately Ubuntu Lucid/Maverick and Fedora 13 all have Sphinx 0.6.6, so if you are running Ubuntu the Drizzle Developers PPA has 1.0 ready for you to install. For Fedora 13 users I have built some RPMs which you can download here:
Sphinx: python-sphinx-1.0.4-1.fc13.noarch.rpm
Sphinx's documentation: python-sphinx-doc-1.0.4-1.fc13.noarch.rpm
Sphinx source: python-sphinx-1.0.4-1.fc13.src.rpm
Now, we just need to write the documentation! We have several of us writing the basics right now, but as always any community help on this would be greatly appreciated.
make html
or
make epub
(yes, you can create epubs!)
There are other formats available. Running 'make sphinx-help' will give you a full list of options available.
To use this you need to be using the latest trunk of Drizzle or the release that is due this week. A minimum python-sphinx 1.0 also needs to be installed (older versions will work, but some output will be incorrect). Unfortunately Ubuntu Lucid/Maverick and Fedora 13 all have Sphinx 0.6.6, so if you are running Ubuntu the Drizzle Developers PPA has 1.0 ready for you to install. For Fedora 13 users I have built some RPMs which you can download here:
Sphinx: python-sphinx-1.0.4-1.fc13.noarch.rpm
Sphinx's documentation: python-sphinx-doc-1.0.4-1.fc13.noarch.rpm
Sphinx source: python-sphinx-1.0.4-1.fc13.src.rpm
Now, we just need to write the documentation! We have several of us writing the basics right now, but as always any community help on this would be greatly appreciated.
Saturday 28 August 2010
MySQL 5.1 Plugins Development Published
MySQL 5.1 has a great feature which not many people know about, that is the fact it can be extended via. the use of plugins. Unfortunately how you go about this is not incredibly well documented. You can search for examples on the internet, dig through the MySQL source code and ask on the forums and you may figure it all out. But doing all this is time consuming and could easily put someone off. So Sergei Golubchik and I have got together to bring you this book which will show you, using examples, how to write your own plugins.We start by explaining the UDF API which has been around for a long time, and then move on to Daemon Plugins, Information Schema Plugins, Full-text Search Plugins and Storage Engine Plugins. Each with usable examples.
MySQL 5.1 Plugins Development has just been published by Packt Publishing and I believe it is well worth a look if you are thinking of extending MySQL.
Wednesday 25 August 2010
Wordpress and Drizzle
I am a firm believer in practising what you preaching so since I am working in Drizzle my blog should work on Drizzle.
After a few small conversions of table schemas this Wordpress blog is now running on a Drizzle database server! I will over the next few weeks migrate the rest of the sites I host as well as other server-side things to use drizzle instead of MySQL.
I also suggest watching this space, I will soon be working on a tool to make such migrations much easier.
After a few small conversions of table schemas this Wordpress blog is now running on a Drizzle database server! I will over the next few weeks migrate the rest of the sites I host as well as other server-side things to use drizzle instead of MySQL.
I also suggest watching this space, I will soon be working on a tool to make such migrations much easier.
Sunday 22 August 2010
LinuxJedi, the new era
I have recently moved on from my roll as a Senior MySQL Support Engineer at Oracle to a Drizzle Developer at Rackspace. As such my previous blog is now sitting idle so this is the replacement.
This new blog may have much less MySQL Cluster content, but I do hope to continue the same quality (I shall leave you to judge on how high that is!) of blog posts.
This new blog may have much less MySQL Cluster content, but I do hope to continue the same quality (I shall leave you to judge on how high that is!) of blog posts.
Subscribe to:
Posts (Atom)