Apr 12, 2013 5:20 PM
ZenSyslog Question-Please assist
-
Like (0)
Hi Guys,
Thank you for taking the time to read this.
I have taken two days to go through the forum and to search for solutions so this is my last resort.
Please note that my knowledge of Zenoss is limited and that I am actually a windows user but i'm willing to learn and find this very exiting!
I have installed Zenoss 4.2.3 on Centos and I am able to monitor a cisco switch and Zhone MXK without any problems.
I want to receive Syslog information from both devices to see when interfaces change state etc and have set them up to forward Syslog messages to the Zenoss server IP.
The Server is not receiving any syslog messages and nothing is being displayed in the "EVENTS" tab or "EVENTS HISTORY".
Is there anything I can check to verify what could be wrong?
ANY help will be greatly appreciated!
Hi Guys,
Can ANYONE assist?
Here is the config file if that will help:
# Config file written out from GUI
duallog False
allowduplicateclears False
uid zenoss
minpriority 6
zenhubpinginterval 30
syslogport 514
watchdog False
eventflushseconds 5.0
hubhost localhost
stats True
monitor localhost
hubusername admin
noreverseLookup False
maxqueuelen 5000
logformat human
hubpassword YW8SkvvJMlkoDbctH4gv
duplicateclearinterval 0
initialHubTimeout 30
logseverity 20
maxlogsize 10240
parsehost False
maxbackuplogs 3
maxparallel 500
logTaskStats 0
logorig False
disable-event-deduplication True
eventflushchunksize 50
listenip 0.0.0.0
hubport 8789
And the log file:
2013-04-15 08:44:35,709 INFO zen.maintenance: Performing periodic maintenance
2013-04-15 08:44:35,710 INFO zen.zensyslog: Counter eventCount, value 15
2013-04-15 08:44:35,710 INFO zen.zensyslog: 2 devices processed (0 datapoints)
2013-04-15 08:44:35,710 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 89 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2013-04-15 08:49:35,718 INFO zen.maintenance: Performing periodic maintenance
2013-04-15 08:49:35,718 INFO zen.zensyslog: Counter eventCount, value 15
2013-04-15 08:49:35,719 INFO zen.zensyslog: 2 devices processed (0 datapoints)
2013-04-15 08:49:35,719 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 89 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2013-04-15 08:54:35,727 INFO zen.maintenance: Performing periodic maintenance
2013-04-15 08:54:35,727 INFO zen.zensyslog: Counter eventCount, value 15
2013-04-15 08:54:35,728 INFO zen.zensyslog: 2 devices processed (0 datapoints)
2013-04-15 08:54:35,728 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 89 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2013-04-15 08:57:25,672 INFO zen.zensyslog: Connecting to localhost:8789
2013-04-15 08:57:25,696 INFO zen.zensyslog: Connected to ZenHub
2013-04-15 08:57:25,719 INFO zen.maintenance: Performing periodic maintenance
2013-04-15 08:57:25,719 INFO zen.zensyslog: Counter eventCount, value 16
2013-04-15 08:57:25,742 INFO zen.zensyslog: 0 devices processed (0 datapoints)
2013-04-15 08:57:25,742 INFO zen.collector.scheduler: Tasks: 1 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2013-04-15 08:57:30,577 INFO zen.zensyslog: Connecting to localhost:8789
2013-04-15 08:57:30,580 INFO zen.zensyslog: Connected to ZenHub
2013-04-15 08:57:30,584 INFO zen.maintenance: Performing periodic maintenance
2013-04-15 08:57:30,584 INFO zen.zensyslog: Counter eventCount, value 17
2013-04-15 08:57:30,585 INFO zen.zensyslog: 0 devices processed (0 datapoints)
2013-04-15 08:57:30,585 INFO zen.collector.scheduler: Tasks: 1 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2013-04-15 09:02:30,594 INFO zen.maintenance: Performing periodic maintenance
2013-04-15 09:02:30,621 INFO zen.zensyslog: Counter eventCount, value 17
2013-04-15 09:02:30,627 INFO zen.zensyslog: 2 devices processed (0 datapoints)
2013-04-15 09:02:30,627 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 1 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2013-04-15 09:07:30,643 INFO zen.maintenance: Performing periodic maintenance
2013-04-15 09:07:30,654 INFO zen.zensyslog: Counter eventCount, value 17
2013-04-15 09:07:30,660 INFO zen.zensyslog: 2 devices processed (0 datapoints)
2013-04-15 09:07:30,661 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 2 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2013-04-15 09:12:30,674 INFO zen.maintenance: Performing periodic maintenance
2013-04-15 09:12:30,680 INFO zen.zensyslog: Counter eventCount, value 17
2013-04-15 09:12:30,686 INFO zen.zensyslog: 2 devices processed (0 datapoints)
2013-04-15 09:12:30,686 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 2 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
Thank you!
It looks like your zensyslog THINKS it is OK. Several things to check:
1) Do make sure that syslog on your Zenoss server is NOT configured to collect syslogs from other devices. You should have the zensyslog process on port UDP/514 (which is what your zensyslog config is saying with the syslogport parameter). If the Operating System syslog is also fighting for that port then it will probably win and zensyslog will not get data. If you have it on your Zenoss server, I find the lsof command very useful to see what processes have control of what TCP/IP ports, eg:
[zenoss@zen42 4.2]$ lsof -i -P | grep 514
python 16370 zenoss 13u IPv4 629745 0t0 UDP *:514
This determines that a python process is connected to UDP/514. To see which process, grep for the process number given in the second field:
[zenoss@zen42 4.2]$ ps -ef | grep 16370
zenoss 3858 20489 0 12:17 pts/4 00:00:00 grep 16370
zenoss 16370 1 0 Apr12 ? 00:02:52 /opt/zenoss/bin/python /opt/zenoss/Products/ZenEvents/zensyslog.py --configfile /opt/zenoss/etc/zensyslog.conf --cycle --daemon --useFileDescriptor=4
[zenoss@zen42 4.2]$
The answer is zensyslog - oh frabjous day!
2) The zensyslog daemon is fairly unique in that it can log the incoming syslog messages - change the logorig parameter from False to True and restart zensyslog. I use this to prove that syslog events are actually arriving at Zenoss. If there is nothing there, the problem is with the configuration in your syslog servers or it is a comms problem (like firewalls not permitting UDP/514).
3) If you are getting syslog message into the logorig file then increase the logging on zensyslog. You can either do this from the GUI or by hacking the file directly (as with all daemon config files). You need to set logseverity to 10 (Debug if you do it through the GUI) and then restart zensyslog.
What is generating your original syslog messages? The zensyslog code can recognise most variants of syslog files but it may be that you have something that it fails to recognise.
You may find my paper, "Event Management for Zenoss Core 4" helpful - http://www.skills-1st.co.uk/papers/jane/zenoss4-events/ . Chapter 4 is on syslog processing.
I have also just announced dates for the 3-day Zenoss Event Management workshop on May 29-31 and June 11-13 - see message/72840#72840 and http://www.skills-1st.co.uk/products/courses/zenoss-events.html .
Cheers,
Jane
Hi Jane,
Thank you so much for your reply I really appreciate it.
Ok, baby steps. This is what I got, do you see anything wrong?
[root@localhost ~]# lsof -i -P | grep 514
python 3159 zenoss 12u IPv4 13481 0t0 UDP *:514
[root@localhost ~]# ps -ef |grep 13481
root 10594 4129 0 13:56 pts/0 00:00:00 grep 13481
Thank you!
Your second command you need to grep for 3159 as that is the process id of the python process. It is the second field that gives you the process id; the 13481 is apparently the "Device" field for lsof - not quite sure what this is.
The problem with lsof as that it only gives the initial command (python) and all the zenoss daemon process lines start with python.
Cheers,
Jane
Hi Jane,
Apologies, here is the info. If I followed your first instructions everything seems fine?
[root@localhost ~]# ps -ef |grep 3159
zenoss 3159 1 0 08:57 ? 00:00:03 /opt/zenoss/bin/python /opt/zenoss/Products/ZenEvents/zensyslog.py --configfile /opt/zenoss/etc/zensyslog.conf --cycle --daemon --useFileDescriptor=4
root 15385 4129 0 17:54 pts/0 00:00:00 grep 3159
I also changed the logorig parameter from False to True and restarted zensyslog.
Here is what I got:
2013-04-15 18:00:07,032 INFO zen.zensyslog: Deleting PID file /opt/zenoss/var/zensyslog-localhost.pid ...
2013-04-15 18:00:07,032 INFO zen.zensyslog: Daemon SyslogDaemon shutting down
2013-04-15 18:00:07,043 ERROR zen.collector.config: Task configLoader configure failed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.
]
2013-04-15 18:00:10,611 INFO zen.zensyslog: Connecting to localhost:8789
2013-04-15 18:00:10,614 INFO zen.zensyslog: Connected to ZenHub
2013-04-15 18:00:10,617 INFO zen.maintenance: Performing periodic maintenance
2013-04-15 18:00:10,617 INFO zen.zensyslog: Counter eventCount, value 21
2013-04-15 18:00:10,618 INFO zen.zensyslog: 0 devices processed (0 datapoints)
2013-04-15 18:00:10,618 INFO zen.collector.scheduler: Tasks: 1 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2013-04-15 18:00:15,672 INFO zen.zensyslog: Connecting to localhost:8789
2013-04-15 18:00:15,678 INFO zen.zensyslog: Connected to ZenHub
2013-04-15 18:00:15,681 INFO zen.maintenance: Performing periodic maintenance
2013-04-15 18:00:15,681 INFO zen.zensyslog: Counter eventCount, value 22
2013-04-15 18:00:15,682 INFO zen.zensyslog: 0 devices processed (0 datapoints)
2013-04-15 18:00:15,682 INFO zen.collector.scheduler: Tasks: 1 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2013-04-15 18:05:15,692 INFO zen.maintenance: Performing periodic maintenance
2013-04-15 18:05:15,693 INFO zen.zensyslog: Counter eventCount, value 22
2013-04-15 18:05:15,702 INFO zen.zensyslog: 1 devices processed (0 datapoints)
2013-04-15 18:05:15,702 INFO zen.collector.scheduler: Tasks: 3 Successful_Runs: 1 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
I configured a ciscto 3750 switch to forward syslog messages to the zenoss server IP and I have done the same with a Zhone server.
Regards
Hein
So you should now have a file called origsyslog.log in $ZENHOME/log, in addition to your zensyslog.log?? Does that have any incoming syslog messages in it?
Cheers,
Jane
Hi Jane,
I located the file in /opt/zenoss/log/origsyslog.log and opened it in windows but there is nothing in it.
PS: Not sure how to open it in centos as you can probably gather.... ;(
What does it mean if it is empty, no logs coming through?
Hmm - sounds like there are no syslog events arriving. If you change directory to $ZENHOME/log and do a ls -l on the file, when was it last updated? Might it have been when you changed the logorig parameter? For example:
[zenoss@zen42 log]$ cd /opt/zenoss/log/
[zenoss@zen42 log]$ ls -l origsyslog.log
-rwxrwxr-x 1 zenoss zenoss 28613171 Apr 16 09:00 origsyslog.log
[zenoss@zen42 log]$ less origsyslog.log
[zenoss@zen42 log]$
The date is the last time the file was updated. the big number (28613171) is the size (which should obviously increase whenever a syslog message is received). less is a unix command to display the contents of a file (space gets the next page, carriage-return gets the next line, if you know some vi editor commands then they work too).
If this file is virtually emptry and not updating when you believe it should be recieving events, then I would certainly focus attention on the syslog systems that are supposed to be forwarding events to Zenoss and at any fiewall that might be in between.
One test I usually start with is to configure the Zenoss server so that his own syslog events are forwarded to Zenoss (have a look at the Event Management paper for an example). If you can get those flowing then you know that zensyslog is capable of reading and interpreting events. and you can focus your attention on the delivery mechanism.
Cheers,
Jane
Hi Jane,
Thank you for your assistance thus far and for not giving up on me!
I ran the following test to verify that the cisco is actually sending out syslog messages.
I downloaded Tftpd32, it has a very simple built in syslog server that you can use. I installed it, ran wireshark and changed the cisco syslog destination from the zenoss server to my laptop interface IP.
I immedietly received syslog messages from the cisco and wireshark confirmed syslog messages sent from the cisco to my laptop!
I logged back into the cisco and changed it back to the zenoss ip address.
Neither Zenoss or wireshark picks up anything...
My laptop also connects to the same switch as the zenoss server with no firewall enabled.
This just keeps getting weirder and weirder... Any more tests to run?
I did what you reccommended and got this:
[root@localhost log]# ls -l origsyslog.log
-rw-r--r-- 1 zenoss zenoss 0 Apr 15 18:00 origsyslog.log
(Not looking good)
And so summerise:
[root@localhost log]# lsof -i -P | grep 514
python 14524 zenoss 49u IPv4 938605 0t0 UDP *:514
[root@localhost log]# ps -ef |grep 938605
root 17724 21799 0 16:49 pts/0 00:00:00 grep 938605
[root@localhost log]# ps -ef |grep 14524
zenoss 14524 1 0 14:24 ? 00:00:01 /opt/zenoss/bin/python /opt/zenoss/Products/ZenEvents/zensyslog.py --configfile /opt/zenoss/etc/zensyslog.conf --cycle --daemon --useFileDescriptor=15
root 17741 21799 0 16:50 pts/0 00:00:00 grep 14524
Good one isn't it?
Are you running wireshark on your laptop?
Is there a firewall running on your Zenoss server? As root, run:
iptables --list -v
I have a line like:
0 | 0 ACCEPT | udp -- any | any | anywhere | anywhere | state NEW udp dpt:syslog |
The Zenoss server's firewall would block stuff before it gets to zensyslog.
Cheers,
Jane
Hey Jane,
Here is the output, not seeying any thing on UDP...
[root@localhost log]# iptables --list -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
3250K 1163M ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED
0 0 ACCEPT icmp -- any any anywhere anywhere
1087 63796 ACCEPT all -- lo any anywhere anywhere
0 0 ACCEPT tcp -- any any anywhere anywhere state NEW tcp dpt:ssh
19730 2364K REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT 3253K packets, 1044M bytes)
pkts bytes target prot opt in out source destination
I think that means that your firewall on your Zenoss server is rejecting incoming syslog messages on udp/514.
Use the System -> Administration -> firewall menus. Under "Other Ports" you may or may not have anything. You need to ensure that UDP/514 is selected here; using the Add button gives you a scrolled list and you should be able to select what you want. I attach a screenshot of my configs.
If you ever want your Zenoss to receive SNMP TRAPs, also add the two entries for TCP and UDP 162.
I believe the first line of your current firewall config is permitting traffic in that is in response to a message from the Zenoss server, so things like SNMP get/set will be allowed in as the conversation starts from Zenoss but both syslog and SNMP TRAPs are messages unsolicited by Zenoss - hence the conversation starts from the outside and you have no firewall rules that allow these message types in.
Cheers,
Jane
Hi Jane,
I just wanted to let you know that......it worked!!!
Thank you so much for your patience with my beginner questions, you should concider becoming a teacher. If you didnt come to the rescue I would never had figured out the problem!
Next step is to sort out the events and how Zenoss displays it but just receiving emails is good for now!
Again, thank you so much!
Grin, I think Jane is a teacher..
Dont forget to mark the question as answered
Follow Us On Twitter »
|
Latest from the Zenoss Blog » | Community | Products | Services Resources | Customers Partners | About Us | ||
Copyright © 2005-2011 Zenoss, Inc.
|
||||||||