Problem
I am running my Zenoss system on CentOS. When I attempted to get zensyslog to work, no messages were received. I determined that both the syslog service and zensyslog were trying to listen on UDP port 514.
I tried changing the zensyslog log port to UDP 10514. The syslog daemon continued to receive external messages on UDP 514. Unfortunately, there is no way to configure syslog to listen on one port and forward messages out on a different port. You can forward messages to another IP, but they will be sent to the same port that syslog is listening to. So, while I was able to receive syslog messages, I could not relay them to zensyslog.
The other option of reconfiguring syslog to listen/forward on UDP 10514 while zensyslog listened on UDP 514 doesn't work for the same reason - External syslog messages reach zensyslog, but local messages processed by syslog don't get to zensyslog.
Solution #1 - rsyslog
This is the simplest solution to the problem - replace sysklogd with rsyslog. At the time of this writing, I am using CentOS 5.3 which only supports rsyslog v2.0.6. Unfortunately, this is a long way away from the current version of rsyslog which supports a lot more features. I recommend you download and unzip the rsyslog v2.0.6 package. The /doc subdirectory contains documentation specific to this version. Most of the online documentation you can find is for newer versions of rsyslog. I wasted a lot of time before I realized that most of the documentation on the Internet contains features not available in v2.0.6. Just stick with the documentation included with the package. So let's get to it!
Step-by-Step
Start out by logging into your system as root. Next, you want to install rsyslog. Don't worry, installing it will not interfere with your current syslog installation.
[root@zenoss ~]# yum install rsyslog
Before the next step, just a little background. Since we want zensyslog to collect all incoming syslog messages, it needs to listen on the default syslog port - UDP 514. No problem - it does that automatically. In order to send local syslog messages to zensyslog (the whole purpose of this article), we will be configuring rsyslog to forward messages to the loopback address on UDP 514. So far so good. However, when you configure rsyslog to forward messages, it automatically opens a port and begins to listen. There is no option to turn it off. By default, it uses UDP 514; however, this can interfere with zensyslog. Since we can't turn it off, we do the next best thing - tell it to listen on a different port.
Edit the /etc/sysconfig/rsyslog file with your favorite editor. Change the line:
SYSLOGD_OPTIONS="-m 0"
to:
SYSLOGD_OPTIONS="-m 0 -r10514 -4"
The -r10514 option tells rsyslog to listen on UDP 10514. That way, it won't interfere with zensyslog. The -4 option restricts rsyslog to IPv4. That one is optional; however, rsyslog will listen on IPv4 and IPv6 ports by default.
OK, now its time to tell rsyslog what to do with syslog messages. RHEL provides a default template and the rsyslog rpm provides a copy of that template for itself. You can read more about customizing how syslog messages are processed at syslog.conf(5). For our purposes we will assume all you want to do is forward messages to zensyslog. The lines below forward any messages of level 'warning' or above to zensyslog. Edit the /etc/rsyslog.conf file and add the following lines:
# Forward all messages of level warning or above to zensyslog
*.warninng @127.0.0.1:514
Turn off the syslog daemon:
[root@zenoss ~]# service syslog stop
Start the rsyslog daemon:
[root@zenoss ~]# service rsyslog start
Check to make sure zensyslog and rsyslog are listening on different ports. Read more about netstat at netstat(8).
[root@zenoss ~]# netstat -lpn | grep 514
udp 0 0 0.0.0.0:514 0.0.0.0:* [pid]/python
udp 0 0 0.0.0.0:10514 0.0.0.0:* [pid]/rsyslogd
Prevent syslog from startup up at bootup and make rsyslog start and bootup:
[root@zenoss ~]# chkconfig --del syslog
[root@zenoss ~]# chkconfig rsyslog on
If you haven't already, verify UDP 514 had been opened in iptables. Read more about iptables at the man page iptables(8). There is also an excellent tutorial at frozentux.net.
[root@zenoss ~]# iptables -L | grep syslog
If you don't get:
ACCEPT udp -- anywhere anywhere udp dpt:syslog
Then enter the following command:
[root@zenoss ~]# iptables --append INPUT --protocol udp --destination-port 514 --jump ACCEPT
Thats it! Go to the last section of this document and perform testing.
Solution #2 - iptables
In this solution, zensyslog is configured to listen on UDP 514 - allowing all incoming syslog messages to be processed by zenoss. Syslog is configured to listen on and forward to UDP 10514 (while it doesn't need to listen, there is no way to enable message forwarding without enabling listening). Local syslog messages are forwarded to any valid external IP address. Iptables is used to catch and redirect all outbound UDP 10514 traffic to the loopback address, changing the port in the process from UDP 10514 to UDP 514. This effectively transmits local syslog messages to zensyslog. The following provides step-by-step instructions for configuring your system.
Step-By-Step
Log into your system as root.
Verify sysklogd and iptables packages are installed
Execute the following commands:
[root@zenoss ~]# yum list installed sysklogd
[root@zenoss ~]# yum list installed iptables
If either command returns: Error: No matching Packages to list, you must install the package using the command:
[root@zenoss ~]# yum install package_name
Verify syslog and iptables are set to automatically start at bootup
Execute the following commands:
[root@zenoss ~]# chkconfig --list syslog
[root@zenoss ~]# chkconfig --list iptables
Each command should return the following:
package_name 0:off 1:off 2:on 3:on 4:on 5:on 6:off
If levels 2 through 5 are not on, execute the following command:
[root@zenoss ~]# chkconfig package_name on
Configure syslog
Syslog listens on and forwards to the port defined as "syslog" in the /etc/services file - which, by default, is the IANA syslog port - UDP 514. Read more about the services file at services(5). Since we want zensyslog to receive all the incoming UDP 514 data, we must change the port that syslog is listening on. I chose UDP 10514.
The only way to change the port used by syslog is to change the "syslog" entry in the /etc/services file. Use your favorite editor to change the "syslog" entry. I like to comment out the original setting and add a new one. My "syslog" entries in the /etc/services file are as follows:
#syslog 514/udp # original syslog setting
syslog 10514/udp # setting to accomodate zenoss
Now we have to tell syslog to send messages to zensyslog. We use the TCP/IP stack as the means of communication. It doesn't matter where we tell syslog to send the messages because iptables will be intercepting the packets and redirecting them to zensyslog. In my case, I selected the random IP address 1.2.3.4. I added the following lines to the end of my /etc/syslog.conf file. Read more about configuring this file at syslog.conf(5).
# Forward all messages with priority warning and above to Zenoss
*.warning @1.2.3.4
Finally, restart the syslog service.
[root@zenoss ~]# service syslog restart
Configure iptables
Iptables is an extremely powerful, but fairly complex IP routing application. At its most basic functionality, it is a firewall. However, it is also much more. Read more about iptables at the man page iptables(8). There is also an excellent tutorial at frozentux.net. The first thing we need to do is add a rule to open UDP 514 so that incoming syslog messages can reach zensyslog. Execute the following command line:
[root@zenoss ~]# iptables --table filter --append INPUT --protocol udp --destination-port 514 --jump ACCEPT
Next we add a rule to capture any outbound UDP 10514 data, change the destination address to the loopback address and the port to UDP 514:
[root@zenoss ~]# iptables --table nat --append OUTPUT --protocol udp --destination-port 10514 --jump REDIRECT --to-port 514
Verify that your rules were accepted by listing the contents of iptables. You are concerned with the parts in red. The other parts may vary a bit.
[root@zenoss ~]# iptables --table filter --list INPUT
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:514
RH-Firewall-1-INPUT all -- anywhere anywhere
[root@zenoss ~]# iptables --table nat --list OUTPUT
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
REDIRECT udp -- anywhere anywhere udp dpt:syslog redir ports 514
Finally, you need to save the iptables configuration:
[root@zenoss ~]# iptables-save > /etc/sysconfig/iptables
Configure zensyslog
If you are still logged in as root, you will want to change to user zenoss to perform these steps.
By default, zensyslog listens on the port defined as "syslog" in the /etc/services file. Sound familiar? We need zensyslog to listen on UDP 514, but in the steps above, we changed the "syslog" setting to UDP 10514. Fortunately, Zenoss lets us manually override the setting in /etc/services by adding an entry in the $ZENHOME/etc/zensyslog.conf file. Use an editor to add the following line to your file:
syslogport 514
Now just restart the zensyslog service:
[zenoss@zenoss ~]$ zensyslog restart
Testing
Use the logger command to write a message to the syslog service. Read more about logger at logger(1).
[zenoss@zenoss ~]$ logger -p user.warning 'Test warning message'
Using the Zenoss web interface, navigate to the localhost device and look in the Events tab - you should see your syslog message.
To verify that you are receiving external syslog messages, you will have to configure a device to push syslog messages to your zenoss machine and then genereate a test message. It should show up in the Events tab of the device.
