Centralised Logging with Rsyslog

Posted by Darren Oakley and Phil Cowans

Modern web applications involve lots of moving parts, distributed on multiple servers. Tracking what is happening and diagnosing issues can be a challenge so a way of centralising your log output is very desirable. This is where a standard UNIX tool such as Syslog can help greatly.

This post describes how to setup a central log server for collating all of your logs in a single location, and configuring your application nodes to send all log messages to it.

Syslog

Syslog is a standard for recording logging messages developed in the 1980s, and used very widely, especially in UNIX environments. The protocol is standardised in RFC 5424, and all mainstream Linux distributions install a Syslog implementation as part of the base system.

Syslog is by far the dominant logging framework for UNIX systems, which is a strong reason for adopting it in preference to other, less widely deployed systems.

The architecture of Syslog separates log message processing from the applications generating the messages through a standalone syslog daemon. For a single host, this is as shown below:

Syslog on a single server

Note that a separate process (logrotate) manages the log files. Typically this involves starting a new file each day, compressing older files and deleting files after a defined retention period.

In more complex environments, Syslog can be configured to forward messages to a remote logging server:

Syslog with a remote, centralised server

This is handled by syslog daemons on each machine communicating with each other, so the application itself only needs to know about the local service, which significantly reduces application complexity compared to some other approaches.

The centralised approach is how Syslog is used for production systems at Macmillan. It has a number of advantages:

  • Logs are all in one place, so debugging a distributed application doesn’t involve accessing multiple machines.
  • Log management infrastructure doesn’t need to be replicated.
  • Logs can be made accessible in a flexible way to all developers without the need for many people to routinely access production application hosts.
  • Logs are physically separate from the application nodes, and therefore provide a more robust audit trail in the case of a security incident.

Setting up Rsyslog

We use RedHat Linux on our servers, but these instructions should also work on CentOS Linux, and the configuration is likely to be very similar using other Linuxes and variants of Syslog.

The default syslog implementation on RedHat systems is Rsyslog - this should already be installed on your servers, but if not, simply run:

sudo yum install rsyslog

On the Central (Receiving) Server

In order for our central server to be setup correctly we need to make a couple of edits to the Rsyslog config file (/etc/rsyslog.conf). First, we must tell Rsyslog to listen for log messages on UDP and TCP ports. Look for the following section in your config file and uncomment the directives as below:

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Then restart Rsyslog:

sudo /etc/init.d/rsyslog restart

Now your Rsyslog daemon will be listening for connections/messages on port 514.

On the Client (Sending) Servers

All you need to do on the client servers is tell Syslog to forward all logs to the central server. This can be achieved by adding the following to the base of the /etc/rsyslog.conf config file:

*.* syslog.example.com

Now all log messages sent via Syslog are also streamed over to our central server syslog.example.com.

Getting a Ruby on Rails Application to Talk to Syslog

To enable a Ruby on Rails application to log via syslog, we use the syslog-logger gem. To get started, it’s as simple as adding gem 'syslog-logger' to your Gemfile, and then editing your config/environments/production.rb file to tell your application to use the gem in the production environment:

# Log to syslog.
syslog_logger = Logger::Syslog.new('myapp', Syslog::LOG_LOCAL0)
config.logger = ActiveSupport::TaggedLogging.new(syslog_logger)

This now sends all of your applications logs to syslog instead of the usual log file. Note that this is configured to log to the local0 syslog facility (facilities are labels attached to the log messages). For custom applications it’s often useful to use local0-7, as these won’t be used by the system and your logs will therefore be easy to filter. This configuration is also setup to tag all of your applications log messages with the string myapp - this is a way of identifying messages coming from your particular application amongst the whole stream of messages in syslog.


Find this post useful, or want to discuss some of the topics?

About The Authors