Caveats with Eventlet

The Stackforge Libra project as with most Openstack based projects is written in Python.  As anyone who has used Python before probably knows, Python has something called a GIL (Global Interpreter Lock).  The GIL basically causes Python to only execute one thread at a time, context switching between the threads.  This means you can't really use threads for performance reasons in Python.

One solution to get a little more performance is to use Eventlet.  Eventlet is a library which uses what is called "Green Threads" and hacks on top of the networking libraries to give a mutli-threaded like feel to an application.  As part of this blogging series for HP's Advanced Technology Group I'll write about some of the things I found out the hard way about Eventlet so hopefully you don't hit them.

What are Green Threads?

Green Threads are basically a way of doing multi-tasking on a single real thread.  They use what is called "Cooperative Yielding" to allow each other to run rather than being explicitly scheduled.  This has the advantage of removing the need for locks in many cases and making asynchronous IO easier.  But they come with caveats which can hurt if you don't know about them.

Threading library patched

One of the first things you typically do with eventlet is "Monkey Patch" standard Python library functions so that the are compatible with cooperative yielding.  For example you want the sleep() function to yield rather than hanging all the green threads up until finished.

The threading library is one of the libraries that is monkey patched and the behaviour suddenly becomes slightly different.  When you try to spawn a thread control will not return back to the main thread until the child thread has finished execution.  So your loop that tries to spawn X threads will suddenly only spawn 1 thread and not spawn the next until that thread has finished.  It is recommended you use Eventlet's green thread calls instead (which will actually work as expected).

Application hangs

Cooperative yielding relies on the library functions being able to yield.  Which means that if you use functions that do not understand this the yielding will not happen and all your green threads (including the main thread) will hang waiting.  Any unpatched system call (such as executing some C/C++ functions) falls into this category.

A common place you can see this is with the MySQLdb library which is a wrapper for the MySQL C connector (libmysqlclient).  If you execute some complex query that will take some time, all green threads will wait.  If your MySQL connection hangs for any reason... well, you are stuck.  I recommend using one of the native Python MySQL connectors instead.

Another place I have seen this is with any library that relies on epoll.  Python-gearman is an example of this.  It seems that Eventlet only patches the select() calls, so anything that uses epoll.poll() is actually blocking with Eventlet.

In summary there are cases where Eventlet can be useful.  But be careful where you are using it or things can grind to a halt really quickly.

Why use double-fork to daemonize?

Something that I have been asked many times is why do Linux / Unix daemons double-fork to start a daemon?  Now that I am blogging for HP's Advanced Technology Group I can try to explain this as simply as I can.

Daemons double-fork

For those who were not aware (and hopefully are now by the title) almost every Linux / Unix service that daemonizes does so by using a double-fork technique.  This means that the application performs the following:

1. The application (parent) forks a child process
2. The parent process terminates
3. The child forks a grandchild process
4. The child process terminates
5. The grandchild process is now the daemon

The technical reason

First I'll give the technical documented reason for this and then I'll break this down into something a bit more consumable.  POSIX.1-2008 Section 11.1.3, "The Controlling Terminal" states the following:

The controlling terminal for a session is allocated by the session leader in an implementation-defined manner. If a session leader has no controlling terminal, and opens a terminal device file that is not already associated with a session without using the O_NOCTTY option (see open()), it is implementation-defined whether the terminal becomes the controlling terminal of the session leader. If a process which is not a session leader opens a terminal file, or the O_NOCTTY option is used on open(), then that terminal shall not become the controlling terminal of the calling process.

The breakdown

When you fork and kill the parent of the fork the new child will become a child of "init" (the main process of the system, given a PID of 1).  You may find others state on the interweb that the double-fork is needed for this to happen, but that isn't true, a single fork will do this.

What we are actually doing is a kind of safety thing to make sure that the daemon is completely detached from the terminal.  The real steps behind the double-fork are as follows:

1. The parent forks the child
2. The parent exits
3. The child calls setsid() to start a new session with no controlling terminals
4. The child forks a grandchild
5. The child exits
6. The grandchild is now the daemon

The reason we do step 4 & 5 is it is possible for the child to regain control of the terminal, but once it has lost control the forked grandchild cannot do this.

Put simply it is a roundabout way of completely detaching itself from the terminal that started the daemon.  It isn't a strict requirement to do this at all, many modern init systems can daemonize a process that will stay in the foreground quite easily.  But it is useful for systems that can't do this and for anything that at some point in time is expected to be run without an init script.

Why VLAIS is bad

At the beginning of the month I was at FOSDEM interfacing and watching talks on behalf of HP's Advanced Technology Group.  Since my core passion is working on and debugging C code I went to several talks on Clang, Valgrind and other similar technologies.  Unfortunately there I several talks I couldn't get into due to the sheer popularity of them but I hope to catch up with video in the future.

One talk I went to was on getting the Linux kernel to compile in Clang.  It appears that there are many changes which are down to Clang being a minimum of C99 compliant and GCC supporting some non-standard language extensions.

The one language extension which stood out for me was called VLAIS which stands for Variable Length Arrays In Structs.  Now, VLAs (Variable Length Arrays) are nothing new in C they have been around a long time.  What we are talking about here are variable length arrays at any point in the struct.  For example:

void foo(int n) {

    struct {

        int x;

        char y[n];

        int z;
    } bar;
}

The char in this struct is what we are talking about here.  This kind of code is used in several places in the kernel, for the most part it is used in encryption algorithms.

How did it get added to GCC?

It came about around 2004 in what appears to be a conversion of a standard from ADA to C.  There is a mailing list post on it here.  It has since been used in the kernel and I can understand the argument that the kernel was never intended to be compiled with anything other than GCC.  But the side of me that likes openness and portability is not so keen.  I suspect the problems that currently plague the kernel for Clang also affect compilers such as Intel's ICC and other native CPU manufacturer's compilers.

So why is it bad?

Well, to start with there is the portability issue.  Any code that uses this will not compile in other compilers.  If you turn on the pedantic C99 flags in GCC it won't compile either (if you aren't doing this then you really should, it shakes out lots of bugs).  Once Linus Torvalds found out about the usage of it in the kernel he called it an abomination and asked for it to be dropped.

Next there is debugging.  I'm not even sure if debuggers understand this and if they do I can well imagine it being difficult to work with, especially if you need to track Z in the example above.

There is the possibility of alignment issues.  Some architectures work much better when the structs are byte aligned by a certain width.  This will be difficult to do with a VLAIS in the middle of the struct.

In general I just don't think it is clean code and if it were me I would be using pointers and allocated memory instead.

Of course this is all my opinion and I'm sure people have other views that I haven't thought of.  Please use the comments box to let me know what you think about VLAIS and its use in the kernel.  You can find out more in the Linux Plumbers Conference 2013 slides.

HAProxy logs byte counts incorrectly

Continuing my LBaaS look back series of blog posts for HP's Advanced Technology Group I am today looking into an issue that tripped us up with HAProxy.

Whilst we were working with HAProxy we naturally had many automated tests going through a Jenkins server.  One such test was checking that the byte count in the logs tallied with bytes received, this would be used for billing purposes.

Unfortunately we always found our byte counts a little off.  At first we found it was due to dropped log messages.  Even after this problem was solved we were still not getting an exact tally.

After some research and reading of code I found out that despite what the manual says the outgoing byte count is measured from the backend server to HAProxy, not the bytes leaving HAProxy.  This means that injected headers are not in the byte count and if HAProxy is doing HTTP compression for you the count will be way off.

My findings were backed by this post from the HAProxy developer.

On average every log entry for us was off by around 30 bytes due to injected headers and cookies.

Given the link above this appears to be something the developer is looking into but I doubt it will be a trivial fix.

Working with Syslog

As part of my transition to HP's Advanced Technology Group I am winding down my contributions to HP Cloud's Load Balancer as a Service project (where I was technical lead).  I thought I would write a few blog posts on things we experienced on this project.

Whilst we were working on the Stackforge Libra project we added a feature that uploads a load balancer's log file to Swift. To do this we store the HAProxy log file using Syslog into a separate file.  The syslog generated file is what Libra uploaded.

Our installation of Libra is on an Ubuntu LTS setup, but the instructions should be very similar for other Linux distributions.

Logging to a New File

Syslog has the facility to capture certain log types and split them into separate files.  You can actually see it doing this in various /var/log/ files.  Having syslog handle this makes it far easier to manage, rotate, etc... than having daemons write to files in their own unique way.

To do this for HAProxy we create the file /etc/rsyslog.d/10-haproxy.conf with the following contents:

$template Haproxy,"%TIMESTAMP% %msg%\n"
local0.* -/mnt/log/haproxy.log;Haproxy
# don't log anywhere else
local0.* ~

In this example we are using /mnt/log/haproxy.log as the log file, our servers have an extra partition there to hold the log files.  The log will be written in a format similar to this:

Dec 10 21:40:20 74.205.152.216:54256 [10/Dec/2013:21:40:19.743] tcp-in tcp-servers/server20 0/0/1075 4563 CD 17/17/17/2/0 0/0

From here you can add a logrotate script called /etc/logrotate.d/haproxy as follows:

/mnt/log/haproxy.log {

    weekly

    missingok

    rotate 7

    compress

    delaycompress

    notifempty

    create 640 syslog adm

    sharedscripts

    postrotate

        /etc/init.d/haproxy reload > /dev/null

    endscript

}

This will rotate weekly, compressing old logs and retaining up to 7 log files.

Syslog Flooding

We soon found a problem, the generated log file was not recording all log entries on a load balancer that was getting hammered.  When looking through the main Syslog file for clues we discovered that the flood protection had kicked in and we were seeing log entries as follows:

Jul 3 08:50:16 localhost rsyslogd-2177: imuxsock lost 838 messages from pid 4713 due to rate-limiting

Thankfully this flood protection can be tuned relatively simply by editing the file /etc/rsyslog.conf and add the following to the end of the file:

$SystemLogRateLimitInterval 2
$SystemLogRateLimitBurst 25000

Then syslog needs to be restarted (reload doesn't seem to apply the change):

sudo service rsyslog restart

After this we found all log entries were being recorded.

How I would do the "Year of Code"

As a member of HP's Advanced Technology Group I'm very interested in government initiatives to get more people into various forms of computer programming.  The current one being talked about in the media is the "Year of Code", the director of this was attacked on Newsnight recently and it felt a little like she was hung out to dry by someone.

I think it is really great that there is an initiative for this, but I worry that we will end up going with the wrong approach that would collapse.  My head was spinning with ideas if I was writing a curriculum for this so I thought I would note them down in this blog post.

Computer Fundamentals

At first I would teach computer fundamentals to children.  You can't even begin to program a computer properly without understanding these.  The three main components of a computer haven't really changed in many years, very much like a car engine, they have just evolved.  By this I mean the CPU, RAM and storage.  The children should be taught these and the rolls they play in a computer.  Even buy an old PC (AMD K6 or early Pentium for example) and get the children to put it together as an exercise.

CPUs run instructions, at a basic level they can only see two numbers at a time and operate on them (yes it is way more complex than that, but we are teaching basic fundamentals here).  Lots of fun animations can be used to illustrate this if it is done properly.  Computer programming languages are compiled or interpreted into this machine language, children need to know this, CPUs don't understand English :)

They should be taught something relative for computer storage (both GB and GiB).  Children will often tell me "my X has Y GB of memory" but they often have no idea what this means they can store or even if that is RAM or flash storage.  What is "Windows" or even an "Operating System"?  This is something that should be taught as well as the alternatives such as "Mac OS" and "Linux" (maybe not teach them that how to use them, but at least that they exist).  Did you know that your Android phone/tablet is also a computer running a form of Linux?  When I gave a talk in a junior school not too long ago many didn't know this, I even showed them Fedora Linux running on my laptop.  The fascination was incredible.

What is dual/quad-core, what is a MHz?  These are all questions that could be answered by the fundamentals section of the curriculum.

At this stage you could teach them IT.  Word processing and spreadsheets haven't changed much since I was in high school in the 90s and I doubt they will change much in the next couple of decades.  We were taught Logo in school as an introduction to programming (I wouldn't call it a programming language but it helps give instructions in a logical order).  Today's equivalent is "Scratch" which would be an acceptable replacement.

The Internet

The Internet can be this big scary thing that will play a large part of their lives for some time to come.  Children should be taught how the Internet is connected together (they don't need to be taught underlying protocols).  They need to be taught not only that it can be unsafe but why/how it can be unsafe.  I believe there are already initiatives to do this in school but with the fundamentals base knowledge they should be able to understand easier what goes on when they request a web page.

Computer Programming

This should not be taught to everyone (beyond Scratch as indicated above).  Only those who show interest in the subject should be taught more advanced subjects.  I wouldn't teach how to play a clarinet to someone who isn't interested in music.  But by all means given children a taster to see if it is something they want to learn.

When I was around 7 years old I taught myself BASIC on a BBC Micro, it was also one of the first machines I learnt assembly language on (I'm not recommending teaching children that, but encourage them if they want to learn).  Although BASIC in that form is hard to find and probably not as relevant there are languages nowadays that could be taught instead.

Last month I was in our Seattle office and during my down time I spent some time with some friends there.  A young lady called Sakura lives there who is only a year older than my eldest son and was telling me all about how she is learning Python.  In my opinion this would be a great starter language.

Combined with a development board such as a Raspberry Pi, a hardware kit and a good Python library set the programming could be extended to basic robotics.  Seeing their work do tangible things in the real world would be a great way of keeping children interested.

HTML is not a programming language.  XML is not a programming language, with any hope XML will be replaced with YAML in the future but that is another blog post.  These two should not be taught in schools but any inquisitive child should be shown things like the w3c schools site.

Summary

We need more computer programmers in the UK.  When I worked as a Technical Lead for HP Cloud's Platform-as-a-Service department I was the only one in the entire department who was UK based, I would have loved to have hired more UK staff.  But not everyone will be / can be a programmer and we should accept this.  We definitely need more IT professionals in the UK, from people who can use word processors and the Internet to people who can program.  The UK seems focused on games programming, negating the fact that there are many other fun things you can do in the computer industry.  When I was 18 and working in a computer shop building and repairing PCs, that was a lot of fun and very hands-on.  Now I find debugging and writing Open Source C tools/libraries a lot of fun and incorporate that in my day job as much as I can.

I do hope the government gets this right and teaches appropriate things whilst keeping it fun and interesting for the children.

These are just my opinions and not necessarily those of HP, please comment with what you think the "Year of Code" initiative should look like.

The importance of backup verification

I have recently moved to HP's Advanced Technology Group which is a new group in HP and as part of that I will be blogging a lot more about the Open Source things I and others in HP work on day to day.  I thought I would kick this off by talking about work that a colleague of mine, Patrick Crews, worked on several months ago.

For those who don't know Patrick, he is a great Devops Engineer and QA.  He will find new automated ways of breaking things that will torture applications (and the Engineers who write them). I don't know if I am proud or ashamed to say he has found many bugs in code that I have written by doing the software equivalent of beating it with a sledgehammer.

Every Devops Engineer worth his salt knows that backups are important, but one thing that is regularly forgotten about is to check whether the backups are good.  A colleague of mine from several years back, Ian Anderson, once told me about the hunt for a good tape archive vendor.  He tested them by getting them to pick a randomly selected tape from the archives and reading it, timing how long it takes to do so.  You would be surprised the vendors who couldn't perform this task, I'd hate to see what would happen in a real emergency.

There was also the case of CouchSurfing which was crippled back in 2006 when after a massive failure they found their backups to be bad.  They eventually rebuilt and is a great site today, but this kind of damage can cost even a small company many thousands of dollars.

The main thing I am trying to stress here is that it is important not just to make backups but also to make sure the backups are recoverable.  This should be done with verification and even fire drills.  There may come a time where you may really need that backup in an emergency and if it isn't there, well you just burnt your house down.

Before I was a member of the Advanced Technology Group I was the Technical Lead for the Load Balancer as a Service project for HP Cloud.  We had a very small team and needed a backup and verification solution that was mostly automated and reliable.  This thing would be hooked up to our paging system and none of my team like being woken at 2AM :)

This is a very crude diagram of the solution Patrick developed:

The solution works as follows:

1. Every X minutes Jenkins will tell the database servers to make a backup
2. The MySQL database servers encrypt that backup and push it up to the cloud file storage (Openstack Swift)
3. Jenkins would then trigger a compute instance (Openstack Nova) build and install MySQL on it
4. The new virtual machine would grab the backup, decrypt it, restore it and run a bunch of tests on the data to see if it is valid
5. If any of the above steps failed send out a page to people

Most of the above uses salt to communicate across the machines.  Have we ever been paged by this system?  Yes, but so far only because step 3 failed, either due to a Nova build fail or once due to a salt version incompatibility.  We have since added some resilience into the system to stop this happening again.

As well as the above there are monthly fire drills to manually test that we can restore from these backups.  We also regularly review the testing procedures to see if we can improve them.

This is going to sound a little strange but sometimes the best Devops Engineers are lazy.  By that I don't mean they don't do any work (they are some of the hardest working people I know), but they will automate everything so they don't have to do a lot of boring manual labour and they hate being woken by pagers at silly hours of the morning.  Some of the best Devops Engineers I know think in these terms :)