Working with Syslog

As part of my transition to HP's Advanced Technology Group I am winding down my contributions to HP Cloud's Load Balancer as a Service project (where I was technical lead).  I thought I would write a few blog posts on things we experienced on this project.

Whilst we were working on the Stackforge Libra project we added a feature that uploads a load balancer's log file to Swift. To do this we store the HAProxy log file using Syslog into a separate file.  The syslog generated file is what Libra uploaded.

Our installation of Libra is on an Ubuntu LTS setup, but the instructions should be very similar for other Linux distributions.

Logging to a New File

Syslog has the facility to capture certain log types and split them into separate files.  You can actually see it doing this in various /var/log/ files.  Having syslog handle this makes it far easier to manage, rotate, etc... than having daemons write to files in their own unique way.

To do this for HAProxy we create the file /etc/rsyslog.d/10-haproxy.conf with the following contents:

$template Haproxy,"%TIMESTAMP% %msg%\n"
local0.* -/mnt/log/haproxy.log;Haproxy
# don't log anywhere else
local0.* ~

In this example we are using /mnt/log/haproxy.log as the log file, our servers have an extra partition there to hold the log files.  The log will be written in a format similar to this:

Dec 10 21:40:20 74.205.152.216:54256 [10/Dec/2013:21:40:19.743] tcp-in tcp-servers/server20 0/0/1075 4563 CD 17/17/17/2/0 0/0

From here you can add a logrotate script called /etc/logrotate.d/haproxy as follows:

/mnt/log/haproxy.log {

    weekly

    missingok

    rotate 7

    compress

    delaycompress

    notifempty

    create 640 syslog adm

    sharedscripts

    postrotate

        /etc/init.d/haproxy reload > /dev/null

    endscript

}

This will rotate weekly, compressing old logs and retaining up to 7 log files.

Syslog Flooding

We soon found a problem, the generated log file was not recording all log entries on a load balancer that was getting hammered.  When looking through the main Syslog file for clues we discovered that the flood protection had kicked in and we were seeing log entries as follows:

Jul 3 08:50:16 localhost rsyslogd-2177: imuxsock lost 838 messages from pid 4713 due to rate-limiting

Thankfully this flood protection can be tuned relatively simply by editing the file /etc/rsyslog.conf and add the following to the end of the file:

$SystemLogRateLimitInterval 2
$SystemLogRateLimitBurst 25000

Then syslog needs to be restarted (reload doesn't seem to apply the change):

sudo service rsyslog restart

After this we found all log entries were being recorded.