Setting up a centralized log server using syslog isn't as hard as many may believe. Whether it's logs from Apache, nginx, email services, or even from your own Python applications having a central log server gives you many benefits:
Reduces disk space usage and disk I/O on core servers that should be busy doing something else. This is especially true if you want to log all queries to your database. Doing this on the same disk as your actual database creates a write for every read and an extra write for every write.
Removes logs from the server in the event of an intrusion or system failure. By having the logs elsewhere you at least have a chance of finding something useful about what happened.
All of your logs are in one place, duh! This makes things like grepping through say Apache error logs across multiple webservers easier than bouncing around between boxes. Any log processing and log rotation can also be centralized which may delay your sysadmin from finally snapping and killing everyone.
In case you aren't terribly familiar with how syslog works, here's a quick primer. Syslog separates out various logs using two items. Facilities and Levels. Here are the standard facilities:
| Facility | Description |
|---|---|
| 0 | Kernel messages |
| 1 | User-level messages |
| 2 | Mail system |
| 3 | System daemons |
| 4 | Security/authorization messages |
| 5 | Messages generated internally by syslogd |
| 6 | Line printer subsystem |
| 7 | Network news subsystem |
| 8 | UUCP subsystem |
| 9 | Clock daemon |
| 10 | Security/authorization messages |
| 11 | FTP daemon |
| 12 | NTP subsystem |
| 13 | Log audit |
| 14 | Log alert |
| 15 | Clock daemon |
| 16 | Local use 0 (local0) |
| 17 | Local use 1 (local1) |
| 18 | Local use 2 (local2) |
| 19 | Local use 3 (local3) |
| 20 | Local use 4 (local4) |
| 21 | Local use 5 (local5) |
| 22 | Local use 6 (local6) |
| 23 | Local use 7 (local7) |
For each facility logs are sent using a particular level, the levels are:
| Level | Name | Description |
|---|---|---|
| 0 | Emergency | System is unusable |
| 1 | Alert | Action must be taken immediately |
| 2 | Critical | Critical conditions |
| 3 | Error | Error conditions |
| 4 | Warning | Warning conditions |
| 5 | Notice | Normal but significant condition |
| 6 | Informational | Informational messages |
| 7 | Debug | Debug-level messages |
So for any given log message you set these two options to give a hint as to where the logs should be directed. For example, if an email server receives a new message it would likely be sent as mail.info and a kernel panic would be sent using kern.emerg.
The receiving syslog server then can be configured to direct log messages of a certain facility and/or log level to various files. For example, a default Ubuntu system has some settings like this:
daemon.* /var/log/daemon.log kern.* /var/log/kern.log mail.* /var/log/mail.log
But you can also do more granular separation for example you might want to log mail.err into a separate file from the main mail logs to make it easier to spot new errors with this:
mail.* /var/log/mail.log mail.err /var/log/mail-errors.log
Configuring the master log server is pretty easy. On Ubuntu the default syslog server is rsyslog and that's what I'll be using as an example here. You'll need to edit /etc/rsyslog.conf and uncomment the UDP module. You could also use the TCP module, but that one binds to all of your interfaces so you will need to restrict access to it with iptables (or some other mechanism) in order to not allow hackers to fill up your disks remotely. So your configuration should now contain these uncommented lines, where 'x' is an internal protected IP address:
$ModLoad imudp $UDPServerAddress x.x.x.x $UDPServerRun 514
And then restart rsyslogd.
systemctl restart rsyslogd.service
Setting up your remote servers is even easier. If you want to send ALL of your logs to the central server it's just a matter of adding one line to the top of /etc/rsyslog.d/50-default.conf. That line is:
*.* @x.x.x.x:514
This will send all logs of any facility and any level to the server. Note that the local syslog will, as configured by default, still log locally. So if you don't want that be sure to remove all of the other configuration in this file.
You can also get fancy here and keep some logs on the local server and only send some things remotely. For most of your custom apps and logs you'll want to be using the LOCAL[0-9] facilities. Let's say we're going to want to centrally log our Python logs and Apache error logs. We'll be using LOCAL0 and LOCAL1 for them respectively. That config would look like:
local0.* @x.x.x.x:514 local1.* @x.x.x.x:514
Keep in mind however that most systems have *.info, *.debug, etc. configurations setup so you might be duplicating your data. If you poke around this file you'll see lots of configurations ending in .none, this instructs rsyslog to not include those facilities in this particular file. So for example, you'd want to edit your /var/log/syslog to resemble this:
*.*;auth,authpriv,local0,local1.none /var/log/syslog
While most applications are easy to setup for use with syslog, here are some pointers for more info on the subject:
We've only really scratched the surface of the features of rsyslog with this setup. You can configure it to do some fairly advanced separation of logs based on the sending host, application name, and other various aspects of the message itself. Refer to the rsyslog documentation for more information on that.
See also syslog-ng.