The importance of backup verification

I have recently moved to HP's Advanced Technology Group which is a new group in HP and as part of that I will be blogging a lot more about the Open Source things I and others in HP work on day to day.  I thought I would kick this off by talking about work that a colleague of mine, Patrick Crews, worked on several months ago.

For those who don't know Patrick, he is a great Devops Engineer and QA.  He will find new automated ways of breaking things that will torture applications (and the Engineers who write them). I don't know if I am proud or ashamed to say he has found many bugs in code that I have written by doing the software equivalent of beating it with a sledgehammer.

Every Devops Engineer worth his salt knows that backups are important, but one thing that is regularly forgotten about is to check whether the backups are good.  A colleague of mine from several years back, Ian Anderson, once told me about the hunt for a good tape archive vendor.  He tested them by getting them to pick a randomly selected tape from the archives and reading it, timing how long it takes to do so.  You would be surprised the vendors who couldn't perform this task, I'd hate to see what would happen in a real emergency.

There was also the case of CouchSurfing which was crippled back in 2006 when after a massive failure they found their backups to be bad.  They eventually rebuilt and is a great site today, but this kind of damage can cost even a small company many thousands of dollars.

The main thing I am trying to stress here is that it is important not just to make backups but also to make sure the backups are recoverable.  This should be done with verification and even fire drills.  There may come a time where you may really need that backup in an emergency and if it isn't there, well you just burnt your house down.

Before I was a member of the Advanced Technology Group I was the Technical Lead for the Load Balancer as a Service project for HP Cloud.  We had a very small team and needed a backup and verification solution that was mostly automated and reliable.  This thing would be hooked up to our paging system and none of my team like being woken at 2AM :)

This is a very crude diagram of the solution Patrick developed:

The solution works as follows:

1. Every X minutes Jenkins will tell the database servers to make a backup
2. The MySQL database servers encrypt that backup and push it up to the cloud file storage (Openstack Swift)
3. Jenkins would then trigger a compute instance (Openstack Nova) build and install MySQL on it
4. The new virtual machine would grab the backup, decrypt it, restore it and run a bunch of tests on the data to see if it is valid
5. If any of the above steps failed send out a page to people

Most of the above uses salt to communicate across the machines.  Have we ever been paged by this system?  Yes, but so far only because step 3 failed, either due to a Nova build fail or once due to a salt version incompatibility.  We have since added some resilience into the system to stop this happening again.

As well as the above there are monthly fire drills to manually test that we can restore from these backups.  We also regularly review the testing procedures to see if we can improve them.

This is going to sound a little strange but sometimes the best Devops Engineers are lazy.  By that I don't mean they don't do any work (they are some of the hardest working people I know), but they will automate everything so they don't have to do a lot of boring manual labour and they hate being woken by pagers at silly hours of the morning.  Some of the best Devops Engineers I know think in these terms :)

Stackforge Libra - Balance your life!

I have been pretty quiet on the blogging front for quite a long time now.  The main reason for this is I have been working very hard on leading a small team which is developing a Stackforge project called 'Libra'.  As you can probably guess from the name, Libra is a Load Balancer as a Service system.  Many of you may not of heard of it but according to Stackalytics it was the 27th biggest project in terms of code contributed during Havana and 2nd biggest in HP (something I am especially proud of because it has been one of the smallest teams in HP Cloud).

It is based on the Atlas API specifications, creates software-based load balancers, is implemented in Python and sits on top of Nova instances rather than working under the cloud.  It also has several unique features which could be converted to run any service on top of a cloud.

Components

Libra consists of several components, the service components are designed to be installed on multiple instances to create a highly available setup:

• API server - A Pecan based API server whose API is based on the Atlas API spec.
• Admin API server - A Pecan based Administrative API server (work in progress) and a whole bunch of modules which automatically maintain the health of the Libra system.
• Pool Manager - A gearman service which will provide the rest of the system with load balancer devices and floating IPs
• Worker - A gearman service sitting on each load balancer to configure it

There is also an easy-to-use command line client called python-libraclient.

The service requires Gearman to communicate between the various components and MySQL as a load balancer configuration data store. Our team has been developing SSL support in Gearman to increase the security of the communications and Libra is probably the first open source project to support this.  Security is a high priority for HP and the Libra team take it very seriously. Libra also supports Galera clusters for MySQL by having built-in auto-failover and recovery as well as commit deadlock detection and transaction retry.  The codebase can support any load balancing software (and in theory hardware) through plugins to the Worker but currently HAProxy is the only plugin being developed.

Auto-Recovery

Libra also has an intelligent auto-recovery system built in. In HP we are currently testing the 4.x release in a few racks that have a flaky networking setup and it has been consistently repairing devices just like a T-1000 repairs bullet holes.  Somehow we have even been able to create usable load balancers from this setup too!

The system is designed to keep a constant pool of hot spare load balancers on standby ready so that one can be provisioned very quickly.  The health of this pool is constantly checked, bad devices in the pool are destroyed and replaces automatically with new ones.

With version 3.x (HP Cloud Private Beta) we are seeing provision times of roughly 300ms.  This has increased slightly for Libra 4.x because it introduces floating IP support (called Virtual IPs in Atlas API).  The floating IP support means that the Libra system can automatically detect when a load balancer has failed, within seconds rebuild the load balancer on a new hot spare and move the IP address accordingly.

Other Uses

The architectural design of Libra is such that it could be used to create any "Platform-as-a-Service" on top of Nova (or any other cloud with minor modifications).  The system can be modified to work with anything by changing the API and giving it a Worker plugin.  This gives Libra codebase the ability to become an eventual framework in the future.

Libra and HP

Libra 3.x is currently installed as our LBaaS Private Beta and an installation of 4.x will be coming in the next few weeks.  We have learnt a lot from customer feedback and have a lot of interesting ideas in the pipeline or in testing (such as Galera cluster load balancers!  Handy for DBaaS).  If you want to find out more about Libra you can see the HP Cloud LBaaS page, look at the developer docs, come chat to us on #stackforge-libra Freenode IRC channel and if you want to give it a spin take a look at our PPA.

We want to be more open with the development process of Libra and are taking steps to ensure that happens by engaging more with the wider community, resource constraints have made this difficult up until now.  We are happy to help anyone who wants to get started playing with or hacking on any part of it.

Libdrizzle 5.1.4 released!

I've just created a release of Libdrizzle 5.1.4 (the BSD licensed C connector for MySQL servers) which for me is one of the most exciting releases to date.

Why?
Not because the amount of features or anything that we have added, in fact this is mostly a minor release with bug fixes.  It is because Brian Aker and I didn't submit the most code for it.  That honour goes to Wim Lewis from The Omni Group who has done a fantastic job fixing up Libdrizzle.

As for the release itself the main fixes revolve around cleaning up code and many fixes to the server-side prepared statement handling.  Behind the scenes Wim has supplied many improvements to the test suite and Brian has setup more platforms to test against in Jenkins.  All this to help improve the code quality of Libdrizzle.

You can download this new release along with older releases from the Libdrizzle downloads page.

New logo

I recently decided it was time for a change, specifically in the logo I have been using for my online identity "LinuxJedi".  Jett Atwood has done some fantastic work designing a new logo based on the crude rambling ideas I had in my head.

You can check out more of Jett's work on her homepage.

MySQL and the forks in the road

There's a lot of activity in the MySQL ecosystem currently. I've written an article for The H which takes a look at MySQL and the various forks and spinoffs to give an overview of where MySQL's community and close competition is currently at.

http://www.h-online.com/open/features/MySQL-and-the-forks-in-the-road-1829242.html

First version of Drizzle Tools for MySQL servers released

Today marks the first release of Drizzle Tools for MySQL servers.  Drizzle Tools aims to be a collection of useful utilities to use with MySQL servers based around the work on the Libdrizzle Redux project.

In this first version there is one utility in the tree called 'drizzle-binlogs'.  If you've seen me talk about this tool before it is because it used to be included in the Libdrizzle 5.1 source but has now been moved here to be developed independently.  For those who haven't 'drizzle-binlogs' is a tool which connects to a MySQL server as a slave, retrieves the binary log files and stores them locally.  This could be used as part of a backup solution or a rapid way to help create a new MySQL master server.

Due to the API changes before the Libdrizzle API became stable Drizzle Tools requires a minimum of Libdrizzle 5.1.3 to be installed.

I wanted to release this sooner but unfortunately most of my time has been taken up with the first release of the project I manage and develop for my day job (HP Cloud's Load Balancer as a Service, more about this in a future blog post).

In the not too distant future there will be more tools included in the Drizzle Tools releases, I have the next one already 50% developed.  In the mean time you can download the first version here.

Using the Libdrizzle Binlog API

Now that we have frozen the 5.1 API of Libdrizzle I can blog about how to use parts of the API.

In this blog post I will cover connecting to a MySQL server and retrieving the binary logs.

First of all we need to connect to the MySQL server

#include <libdrizzle-5.1/libdrizzle.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <inttypes.h>

int main(void)
{
  drizzle_st *con;
  drizzle_binlog_st *binlog;
  drizzle_return_t ret;
 
  con= drizzle_create("localhost", 3306, "user", "pass", "", NULL);
  ret= drizzle_connect(con);
  if (ret != DRIZZLE_RETURN_OK)
  {
    printf("Could not connect to server: %s\n", drizzle_error(con));
    return EXIT_FAILURE;
  }

We are connecting to a local server on port 3306 with the username "user" and password "pass".  We don't need a default database so we leave that empty and set the options to NULL.

After we are connected to the MySQL server we need to initialize the binary log transfer.

  binlog= drizzle_binlog_init(con, binlog_event, binlog_error, NULL, true);

This states that we want an event to call function called "binlog_event" and an error to call a function called "binlog_error".  We could pass a pointer data to callback functions but for this example we have set it to "NULL".  The final parameter specifies whether or not we want to verify the checksums in MySQL 5.6 binary logs.

Then we need to start the binary log transfer.

  ret= drizzle_binlog_start(binlog, 0, "", 0);
  if (ret != DRIZZLE_RETURN_EOF)
  {
    return EXIT_FAILURE;  }
  drizzle_con_quit(con);
  return EXIT_SUCCESS;
}

This states that we want to start the binary log at the first position of the first file using server ID 0.  With server ID 0 the MySQL server will end the transfer at the end of the last binary log file.  The function will return at the end of the transfer with DRIZZLE_RETURN_EOF or an error code.

All we need now are error callback and event callback.  So lets start with the error callback.

void binlog_error(drizzle_return_t ret, drizzle_st *con, void *context)
{
  (void) context;
  if (ret != DRIZZLE_RETURN_EOF)
  {
    printf("Error retrieving binlog: %s\n", drizzle_error(con));
  }
}

Finally we need to event callback.

void binlog_event(drizzle_binlog_event_st *event, void *context)
{
  (void) context;
  uint32_t length;
  printf("Timestamp: %"PRIu32"\n", drizzle_binlog_event_timestamp(event));
  printf("Type: %"PRIu8"\n", drizzle_binlog_event_type(event));
  printf("Server-id: %"PRIu32"\n", drizzle_binlog_event_server_id(event));
  printf("Next-pos: %"PRIu32"\n", drizzle_binlog_event_next_pos(event));
  length= drizzle_binlog_event_length(event);
  printf("Length: %"PRIu32"\n", length);
  data= drizzle_binlog_event_data(event);
  printf("Data: 0x");
  for (i=0; i<length; i++)
    printf("%02X ", data[i]);
  printf("\n\n");
}

On every event received this function is called and will basically spit out the event details.

For a further example see the source of the drizzle_binlogs tool in the Drizzle Tools tree.