Centralised Logging with Rsyslog
Modern web applications involve lots of moving parts, distributed on multiple servers. Tracking what is happening and diagnosing issues can be a challenge so a way of centralising your log output is very desirable. This is where a standard UNIX tool such as Syslog can help greatly.
This post describes how to setup a central log server for collating all of your logs in a single location, and configuring your application nodes to send all log messages to it.
Syslog
Syslog is a standard for recording logging messages developed in the 1980s, and used very widely, especially in UNIX environments. The protocol is standardised in RFC 5424, and all mainstream Linux distributions install a Syslog implementation as part of the base system.
Syslog is by far the dominant logging framework for UNIX systems, which is a strong reason for adopting it in preference to other, less widely deployed systems.
The architecture of Syslog separates log message processing from the applications generating the messages through a standalone syslog daemon. For a single host, this is as shown below:
Note that a separate process (logrotate) manages the log files. Typically this involves starting a new file each day, compressing older files and deleting files after a defined retention period.
In more complex environments, Syslog can be configured to forward messages to a remote logging server:
This is handled by syslog daemons on each machine communicating with each other, so the application itself only needs to know about the local service, which significantly reduces application complexity compared to some other approaches.
The centralised approach is how Syslog is used for production systems at Macmillan. It has a number of advantages:
- Logs are all in one place, so debugging a distributed application doesn’t involve accessing multiple machines.
- Log management infrastructure doesn’t need to be replicated.
- Logs can be made accessible in a flexible way to all developers without the need for many people to routinely access production application hosts.
- Logs are physically separate from the application nodes, and therefore provide a more robust audit trail in the case of a security incident.
Setting up Rsyslog
We use RedHat Linux on our servers, but these instructions should also work on CentOS Linux, and the configuration is likely to be very similar using other Linuxes and variants of Syslog.
The default syslog implementation on RedHat systems is Rsyslog - this should already be installed on your servers, but if not, simply run:
On the Central (Receiving) Server
In order for our central server to be setup correctly we need to make a couple of edits to the Rsyslog config file (/etc/rsyslog.conf
). First, we must tell Rsyslog to listen for log messages on UDP and TCP ports. Look for the following section in your config file and uncomment the directives as below:
Then restart Rsyslog:
Now your Rsyslog daemon will be listening for connections/messages on port 514.
On the Client (Sending) Servers
All you need to do on the client servers is tell Syslog to forward all logs to the central server. This can be
achieved by adding the following to the base of the /etc/rsyslog.conf
config file:
Now all log messages sent via Syslog are also streamed over to our central server syslog.example.com
.
Getting a Ruby on Rails Application to Talk to Syslog
To enable a Ruby on Rails application to log via syslog, we use the
syslog-logger gem. To get started, it’s as simple as adding gem
'syslog-logger'
to your Gemfile
, and then editing your config/environments/production.rb
file to tell your
application to use the gem in the production environment:
This now sends all of your applications logs to syslog instead of the usual log file. Note that this is configured to
log to the local0
syslog facility (facilities are labels
attached to the log messages). For custom applications it’s often useful to use local0-7
, as these won’t be used by
the system and your logs will therefore be easy to filter. This configuration is also setup to
tag all of your applications log messages with the string myapp
- this
is a way of identifying messages coming from your particular application amongst the whole stream of messages in syslog.
Find this post useful, or want to discuss some of the topics?
About The Authors
-
Darren Oakley
Visit Darren's personal website and blog