Showing posts with label FOSDEM. Show all posts
Showing posts with label FOSDEM. Show all posts

Tuesday, 11 February 2014

Why VLAIS is bad

At the beginning of the month I was at FOSDEM interfacing and watching talks on behalf of HP's Advanced Technology Group.  Since my core passion is working on and debugging C code I went to several talks on Clang, Valgrind and other similar technologies.  Unfortunately there I several talks I couldn't get into due to the sheer popularity of them but I hope to catch up with video in the future.

One talk I went to was on getting the Linux kernel to compile in Clang.  It appears that there are many changes which are down to Clang being a minimum of C99 compliant and GCC supporting some non-standard language extensions.

The one language extension which stood out for me was called VLAIS which stands for Variable Length Arrays In Structs.  Now, VLAs (Variable Length Arrays) are nothing new in C they have been around a long time.  What we are talking about here are variable length arrays at any point in the struct.  For example:

void foo(int n) {
    struct {
        int x;
        char y[n];
        int z;
    } bar;
}

The char in this struct is what we are talking about here.  This kind of code is used in several places in the kernel, for the most part it is used in encryption algorithms.

How did it get added to GCC?


It came about around 2004 in what appears to be a conversion of a standard from ADA to C.  There is a mailing list post on it here.  It has since been used in the kernel and I can understand the argument that the kernel was never intended to be compiled with anything other than GCC.  But the side of me that likes openness and portability is not so keen.  I suspect the problems that currently plague the kernel for Clang also affect compilers such as Intel's ICC and other native CPU manufacturer's compilers.

So why is it bad?


Well, to start with there is the portability issue.  Any code that uses this will not compile in other compilers.  If you turn on the pedantic C99 flags in GCC it won't compile either (if you aren't doing this then you really should, it shakes out lots of bugs).  Once Linus Torvalds found out about the usage of it in the kernel he called it an abomination and asked for it to be dropped.

Next there is debugging.  I'm not even sure if debuggers understand this and if they do I can well imagine it being difficult to work with, especially if you need to track Z in the example above.

There is the possibility of alignment issues.  Some architectures work much better when the structs are byte aligned by a certain width.  This will be difficult to do with a VLAIS in the middle of the struct.

In general I just don't think it is clean code and if it were me I would be using pointers and allocated memory instead.

Of course this is all my opinion and I'm sure people have other views that I haven't thought of.  Please use the comments box to let me know what you think about VLAIS and its use in the kernel.  You can find out more in the Linux Plumbers Conference 2013 slides.

Saturday, 2 February 2013

Developing Libdrizzle

This weekend I am supposed to be giving a talk at FOSDEM on Libdrizzle.  Unfortunately my kids and I all fell ill on Thursday (my wife appears to be immune) so I had to cancel my plans (infecting 5000 people didn't seem wise :)

Instead I am writing this blog post about Libdrizzle and my part in it which covers some of what I was going to talk about.

History of Libdrizzle

Libdrizzle started out as a from-scratch C connector for Drizzle and MySQL originally created by Eric Day.  It was designed to be high performance and use common standards to make it easy to work on.  In the summer of 2010 it was merged into the main Drizzle tree where development has been focused.  There were several attempts to split it out again but until now none were truly successful.

For a few years Brian Aker and I have been randomly discussing things we could have done to make the API easier for developers and users.  As well as features we could possibly add in the future.  Fast forward to Summer 2012 and I decided to do something about this.

The Dawn of Libdrizzle Redux

Around July 2012 I took the Libdrizzle 1.0 code from Drizzle trunk and got it to compile separately using CMake.  I then had the groundwork to work on some of the things we had talked about.  I called this the Libdrizzle Redux project since we were bringing new life to Libdrizzle, primarily as a MySQL Connector.

Initially this was going to be version 3.0 but it was pointed out by Henrik that 3.0 and 4.0 were used as library versions in certain packages so we went straight to 5.0.

After getting to the "It Compiles!" state the first thing I did was strip out the server side API.  I had spoken to many people over the years about the connector and each one only used the client side API, not the server side.  The other thing that was made plainly clear was the API was too complex.  There were many functions that were confusing as to whether the library or the application was supposed to allocate and free objects.  So this was massively simplified very quickly.

Libdrizzle Today

Between Brian and I there have been massive changes to the Libdrizzle code and we now have the basis for a regression suite which has already found several problems with the original code.  Just off the top of my head:
  • There is a new Prepared Statement API
  • There is a new Binary Log Retrieval API (in 5.1.3 this will become a callback API) which is already MySQL 5.6 checksum compatible
  • Massively simplified API, the number of things needed to setup a connection and execute a query have been reduced
  • Non-blocking API has been made more stable
  • Windows support (via. MinGW)
  • Buffer and state system massively improved
  • CMake has been replace with DDM4
  • We compile in C++ with many warnings switch on (C++ compiler tends to find more problems than the normal C compiler)
  • 106 revisions/merges in the BZR repo.  Most of this since November 2012.
Last Sunday Brian and I had a hacking day in Seattle, we worked non-stop on Libdrizzle for nearly 12 hours, I doubt there were many sections of code that were not touched that day.

Why I Develop Libdrizzle

My day job is the Project Manager on Load Balancer as a Service at HP Cloud (you should hear a lot more about that in coming months).  Whilst this is great I don't get to touch and debug a whole lot of code any more.  So Libdrizzle is something I do in my spare time to keep my brain fully active.

I also like to think that the code will be useful to someone.  I'm actually a fan of the BSD license in this context.  It means commercial code can easily statically compile with Libdrizzle which in-turn means more MySQL Server/Percona Server/MariaDB Server adoption and more support contracts for Oracle/Percona/SkySQL/etc...

Even if no one uses the code I still find it a fun challenge and something I will continue for as long as I can.

The Future of Libdrizzle

In the next few days we will be releasing Libdrizzle 5.1.3, this will contain all the work that was done in Seattle (and my flights to/from Seattle) including a few nice API improvements.  Along with this I will create the first release of Drizzle Tools.

Drizzle Tools is a project which contains utilities for use with a MySQL server, at the moment the only one included is drizzle_binlogs which connects to a remote MySQL server as a slave and downloads the binary logs to local files as a back.  It also has a continuous mode which effectively gives an up-to-the-second backup of the data.  I have the code half-written for the next tool which will not make the first release but should be useful to many people.

Of course anyone is welcome to come hack on Librizzle and Drizzle Tools, file bugs, ask questions, etc...  The more knowledge share the better :)

There will be more posts soon showing how to use Libdrizzle and drizzle_tools as well as announcing the new features, so watch this space!

Friday, 18 January 2013

Libdrizzle 5.1.2 released and Libdrizzle at FOSDEM

Libdrizzle is getting better and better with every release, and to reflect this we have two announcements to make.

First of all, Libdrizzle 5.1.2 has been released.  This release has several major bug fixes and performance enhancements.  Changes are as follows:
  • Non-blocking Windows connections are now more stable
  • Improvements to Windows building
  • Unix Socket connections are now more stable
  • Memory allocation/freeing cleanups (for performance and ease of code)
  • Network packet buffer now much more flexible
  • Many performance improvements (the bundled drizzle_binlogs tool is now around 10x faster on my i7 laptop)
  • Several other minor bug fixes
API chages:
  • drizzle_query_str() has been removed, drizzle_query() with a 0 byte length parameter now does the same thing.
As always you can download the tarball and documentation from the Libdrizzle Launchpad Page.

I'm also going to be giving a talk entitled "Libdrizzle Redux, the next-gen MySQL compatible connector" on Sunday 3rd Feb in the MySQL devroom at FOSDEM.  This will be a great chance to find out more about the work we have been doing on the connector and how it can benefit you.

Thursday, 15 November 2012

Upcoming MySQL Conferences

I may not work on many MySQL related things any more but there are some MySQL ecosystem events that I am participating in.

First there is Percona Live London.  This event is on the 3rd and 4th of December at The Millennium Gloucester Conference Centre.  At this event I will be giving a talk on MySQL Compatible Open Source Connectors.  This will cover alternatives to the standard connectors and how to use them, it will be useful if you find the licensing behind the standard connectors prohibiting.  There is also a 40% off discount code you can use when registering, simply enter the promo code "Come2mytalk".

On the 2nd and 3rd of February is FOSDEM.  At this event there is a "MySQL and Friends" dev room on the Sunday.  I am on the committee overseeing the talks submitted to this room and am looking forward to some great talk submissions.  One of the great things about FOSDEM is it is completely free and no registration is required.

I'm looking forward to seeing everyone related to the MySQL ecosystem at these events!