Mydumper 0.5.1 released

After much feedback from the community who have been using mydumper I have created the first bugfix release of the 0.5 series of mydumper.  Changes in this release are as follows:

• Fix MySQL 5.0 compile issue


• Make the metadata file visible (after muliple requests)


• Add --no-lock option to mydumper


• First --daemon snapshot is now at execution instead of the first timed interval


• Fix CentOS 5.x compile issue (also affected Mac OSX)


• Fix libmysqlclient search bug


• Add cppcheck target


• Fix errors flaged by cppcheck


• Add option to turn off docs build


• Add status output at end of CMake

To obtain this release you can download it from Launchpad.

Many thanks to everyone who has been trying it and giving feedback.

An apology to MySQL@Oracle

About a week ago I appeared to have started an argument which has now spanned several blog post comments, 39 Google+ comments and two other blog posts.  So the first thing I want to do is apologise to the MySQL@Oracle staff (and Sheeri).  I was disagreeing on the use of 'The' whilst the real problem is my personal understanding the meaning of the word/name/trademark 'MySQL'.  You guys are right and I am wrong.  To correct this I'll refer to what I used to call the "MySQL Ecosystem" as the Open Database Community until someone has a cool name for it.

The arguments were so severe that last week I was even thinking about leaving the Open Database Community.  Apart from Drizzle, mydumper and a few other things I don't contribute as much useful content I used to now anyway.  I would need about another 20 hours in every day to do as much as I would like to do.  But for now I will just stay more in the background just pushing out code to anyone who wants it.

In reply to Herik's post, please don't blame Monty Taylor, this was all my fault.  Also I never intended for the argument to be hidden (or exist at all) on Google+.  Someone at Oracle pointed out that it was wrong to make any of those comments public, so they will stay hidden for now.

mydumper, now with continuous backup mode!

It has happened to the best of us.  Our MySQL servers are running nicely when the tea-lady in the data centre trips over a wire, knocks a cup hot of java down the UPS batteries and fries every server in the data centre.  OK, so that doesn't happen very often, but there have been cases where data centre failures have physically damaged servers.

So, if this happens with your MySQL server, what do you do?  You can setup new servers in a new data centre and hope you have an up-to-date backup.  But unless you had remote slaves these backups probably won't be as current as they could be.  When you get to the point where you have many servers keeping slaves for every one just for backups can be difficult to maintain.  Many DBAs have different backup strategies to cover this potential problem.

This is where the new mydumper 0.5.0 comes in.  It has a new 'daemon' mode which lets it run in the background.  When in daemon mode it will take multi-threaded consistent snapshots on a regular basis (60 minutes by default) whilst also pretending to be a MySQL slave and continuously retrieving a local copy of binary logs.  This can, in-theory, give an up-to-the-second backup of your server.  Right up until the point where you need to fire the tea-lady.  As an example you can use it in this mode as follows to make snapshots every 2 hours and constantly retrieve the binary logs:

mydumper --daemon --snapshot-interval=120 --logfile=mydumper.log --user=mydumper_user
--password=mydumper_password --host=mysql_server

A maximum of 2 snapshots are retained, one for the previous snapshot and one for the current one in case a failure happens during the current snapshot.  A symbolic link inside the dump directory will always point to the last good snapshot.  Each snapshot contains the binary log positions from that snapshot so at any given time the snapshot can be loaded in using myloader (a multi-threaded restoration tool bundled with mydumper) and then the log applied from that point using mysqlbinlog.  In later versions I plan to have myloader handle the binary logs too.

Mydumper 0.5.0 is currently considered in an 'alpha' stage (0.2.3 is the latest stable version but this will only take single snapshots).  This means there are probably bugs I haven't spotted yet and there is still a little bit of work to do (such as better myloader support for the daemon mode snapshots).  You can download it from Launchpad here.

Mydumper 0.2.3 released!

Today marks the release of mydumper 0.2.3.  Mydumper is a multi-threaded high-performance data dumper (and loader) for MySQL and Drizzle written in C. This is a minor bugfix release whilst I work on the upcoming 0.5 version. The end goal here will be that there is always a 'stable' and 'development' version. 0.2 will be the first stable version and will only have bug fixes. 0.5 will be the first 'development' version where the next planned set of new features will hit.

So, the changes since 0.2.2 are:

• Drizzle support now fully works again


• Fixes so mydumper compiles in FreeBSD (thanks to Kirill A. Korinskiy)

If you wish to try this version the source can be downloaded here.

Mydumper now with MyISAM consistent snapshots!

Mydumper 0.2.2 has been released today with a number of fixes and new features.  The one that most people have been asking for is consistent snapshots for non-InnoDB tables (such as MyISAM).  We have been able to achieve this without locking the database for the entire backup using the following method:

1. Flush tables with read lock (and start transaction with consistent snapshot on all threads)


2. Dump non-InnoDB


3. Start InnoDB dump


4. When non-InnoDB dump has finished (whilst InnoDB is dumping) unlock tables


5. ...


6. Profit

I have also started work on Drizzle support this week.  The Drizzle support is not entirely complete mostly down to Drizzle's handling of SHOW TABLE STATUS and I expect to have it fully working next week.

Here is the list of changes since 0.2.1:

• Consistent snapshots for non-InnoDB (non-transactional) tables


• Fix --binlogs breaking consistent snapshots


• (very) Minor performance improvement to mydumper


• Initial support for Drizzle


• Add --verbose option


• Fix multiple ';' at end of file


• Fix myloader not closing files


• Improve myloader error messages


• Several fixes to documentation building


• Make myloader tell mysqld to not binlog imports by default (new option --enable-binlog to log them)


• Add --database to myloader for single database import to a different database


• Change mydumper's --schemas to --no-schemas (--schemas is now default)


• Add verbose messages (with --verbose=3)


• Fix memory leaks in myloader

You can download mydumper 0.2.2 from Launchpad or by clicking here.

Many thanks to everyone who has been testing mydumper and giving feedback.  Your comments and suggestions go a long way to improving mydumper.

Mydumper now with myloader!

It has only been a few days since the 0.2.0 release of mydumper but there have been some big changes since then.  I will try and go over them all here.

Mydumper 0.2.1

Mydumper 0.2.1 has been released today, many thanks for all those who have been testing the trunk source, the feedback has gone a long way to making fixes and improvements to mydumper.  You can download the source for it here.

Myloader

A few days ago Mark Callaghan asked about restoring mydumper backups.  There is of course the great 'myimporter' tool written by Mikael Fridh, but I wanted something that could integrate into the mydumper source better.  So, I have been busy hacking for the last few days and the big news in today's release is the addition of 'myloader'.  Myloader is a high-performance, multi-threaded tool written in C designed to read mydumper backups and apply them in parallel.

There are still many features I wish to add to it, but is good for basic restorations.

New Website

Last night I knocked together a site for mydumper.  There is not much in the way of content yet, but you can take a look at http://www.mydumper.org/

Mikael FridhMikael Fridh

MySQL Data Dumper 0.2.0 released!

A couple of years ago Domas Mituzas created a tool which could basically be thought of as basically a lightweight multi-threaded mysqldump.  By this I mean it can retrieve data from multiple tables simultaneously and can even break a table down into parts for simultaneous retrieval.  Sometime around 2010 I started hacking on mydumper too but stopped whilst working at Rackspace (Drizzle was way more than full-time for me).

Back when Domas first blogged about it he was managing to dump his sample data over 10x faster than mysqldump!

Since the 0.1 series Domas has fixed a lot of problems and since starting at SkySQL I have finished a lot of work that I started over a year ago.

Today sees the first release of the 0.2 series of mydumper, called 0.2.0.  It has many changes over the last 0.1 release:

• Better error handling


• Many bug fixes


• Documentation


• Binary log dumping


• Compression protocol support


• Schema dumps

My personal favourite fetaure I added here is the binary log dumping.  Mydumper can connect to a MySQL server and retrieve all the binary logs in parallel whilst also retrieving the table data.

Many more features are already in development including additional features on the binary log dumping but for those who want to try it now you can obtain the source from Launchpad here.