Archived community.zenoss.org | full text search
Skip navigation
Up to Discussions in zenoss-users
1 2 Previous Next 3708 Views 16 Replies Latest reply: Apr 22, 2013 8:50 AM by jcurry RSS
heinds Rank: White Belt
heinds
16 posts since
Apr 10, 2013
Currently Being Moderated

Apr 12, 2013 5:20 PM

ZenSyslog Question-Please assist

Hi Guys,

 

Thank you for taking the time to read this.

 

I have taken two days to go through the forum and to search for solutions so this is my last resort.

 

Please note that my knowledge of Zenoss is limited and that I am actually a windows user but i'm willing to learn and find this very exiting!

 

I have installed Zenoss 4.2.3 on Centos and I am able to monitor a cisco switch and Zhone MXK without any problems.

 

I want to receive Syslog information from both devices to see when interfaces change state etc and have set them up to forward Syslog messages to the Zenoss server IP.

 

The Server is not receiving any syslog messages and nothing is being displayed in the "EVENTS" tab or "EVENTS HISTORY".

 

Is there anything I can check to verify what could be wrong?

 

ANY help will be greatly appreciated!

  • heinds Rank: White Belt
    heinds
    16 posts since
    Apr 10, 2013
    Currently Being Moderated
    1. Apr 15, 2013 3:16 AM (in response to heinds)
    Re: ZenSyslog Question-Please assist

    Hi Guys,

     

    Can ANYONE assist?

     

    Here is the config file if that will help:

     

     

     

    # Config file written out from GUI

    duallog False

    allowduplicateclears False

    uid zenoss

    minpriority 6

    zenhubpinginterval 30

    syslogport 514

    watchdog False

    eventflushseconds 5.0

    hubhost localhost

    stats True

    monitor localhost

    hubusername admin

    noreverseLookup False

    maxqueuelen 5000

    logformat human

    hubpassword YW8SkvvJMlkoDbctH4gv

    duplicateclearinterval 0

    initialHubTimeout 30

    logseverity 20

    maxlogsize 10240

    parsehost False

    maxbackuplogs 3

    maxparallel 500

    logTaskStats 0

    logorig False

    disable-event-deduplication True

    eventflushchunksize 50

    listenip 0.0.0.0

    hubport 8789

     

    And the log file:

     

    2013-04-15 08:44:35,709 INFO zen.maintenance: Performing periodic maintenance

    2013-04-15 08:44:35,710 INFO zen.zensyslog: Counter eventCount, value 15

    2013-04-15 08:44:35,710 INFO zen.zensyslog: 2 devices processed (0 datapoints)

    2013-04-15 08:44:35,710 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 89 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

    2013-04-15 08:49:35,718 INFO zen.maintenance: Performing periodic maintenance

    2013-04-15 08:49:35,718 INFO zen.zensyslog: Counter eventCount, value 15

    2013-04-15 08:49:35,719 INFO zen.zensyslog: 2 devices processed (0 datapoints)

    2013-04-15 08:49:35,719 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 89 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

    2013-04-15 08:54:35,727 INFO zen.maintenance: Performing periodic maintenance

    2013-04-15 08:54:35,727 INFO zen.zensyslog: Counter eventCount, value 15

    2013-04-15 08:54:35,728 INFO zen.zensyslog: 2 devices processed (0 datapoints)

    2013-04-15 08:54:35,728 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 89 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

    2013-04-15 08:57:25,672 INFO zen.zensyslog: Connecting to localhost:8789

    2013-04-15 08:57:25,696 INFO zen.zensyslog: Connected to ZenHub

    2013-04-15 08:57:25,719 INFO zen.maintenance: Performing periodic maintenance

    2013-04-15 08:57:25,719 INFO zen.zensyslog: Counter eventCount, value 16

    2013-04-15 08:57:25,742 INFO zen.zensyslog: 0 devices processed (0 datapoints)

    2013-04-15 08:57:25,742 INFO zen.collector.scheduler: Tasks: 1 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

    2013-04-15 08:57:30,577 INFO zen.zensyslog: Connecting to localhost:8789

    2013-04-15 08:57:30,580 INFO zen.zensyslog: Connected to ZenHub

    2013-04-15 08:57:30,584 INFO zen.maintenance: Performing periodic maintenance

    2013-04-15 08:57:30,584 INFO zen.zensyslog: Counter eventCount, value 17

    2013-04-15 08:57:30,585 INFO zen.zensyslog: 0 devices processed (0 datapoints)

    2013-04-15 08:57:30,585 INFO zen.collector.scheduler: Tasks: 1 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

    2013-04-15 09:02:30,594 INFO zen.maintenance: Performing periodic maintenance

    2013-04-15 09:02:30,621 INFO zen.zensyslog: Counter eventCount, value 17

    2013-04-15 09:02:30,627 INFO zen.zensyslog: 2 devices processed (0 datapoints)

    2013-04-15 09:02:30,627 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 1 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

    2013-04-15 09:07:30,643 INFO zen.maintenance: Performing periodic maintenance

    2013-04-15 09:07:30,654 INFO zen.zensyslog: Counter eventCount, value 17

    2013-04-15 09:07:30,660 INFO zen.zensyslog: 2 devices processed (0 datapoints)

    2013-04-15 09:07:30,661 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 2 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

    2013-04-15 09:12:30,674 INFO zen.maintenance: Performing periodic maintenance

    2013-04-15 09:12:30,680 INFO zen.zensyslog: Counter eventCount, value 17

    2013-04-15 09:12:30,686 INFO zen.zensyslog: 2 devices processed (0 datapoints)

    2013-04-15 09:12:30,686 INFO zen.collector.scheduler: Tasks: 4 Successful_Runs: 2 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

     

     

    Thank you!

  • jcurry ZenossMaster
    jcurry
    1,021 posts since
    Apr 15, 2008
    Currently Being Moderated
    2. Apr 15, 2013 7:32 AM (in response to heinds)
    Re: ZenSyslog Question-Please assist

    It looks like your zensyslog THINKS it is OK.  Several things to check:

     

    1) Do make sure that syslog on your Zenoss server is NOT configured to collect syslogs from other devices.  You should have the zensyslog process on port UDP/514 (which is what your zensyslog config is saying with the syslogport parameter).  If the Operating System syslog is also fighting for that port then it will probably win and zensyslog will not get data.  If you have it on your Zenoss server, I find the lsof command very useful to see what processes have control of what TCP/IP ports, eg:

     

    [zenoss@zen42 4.2]$ lsof -i -P | grep 514

    python  16370 zenoss   13u  IPv4  629745      0t0  UDP *:514

     

    This determines that a python process is connected to UDP/514.  To see which process, grep for the process number given in the second field:

     

    [zenoss@zen42 4.2]$ ps -ef | grep 16370

    zenoss    3858 20489  0 12:17 pts/4    00:00:00 grep 16370

    zenoss   16370     1  0 Apr12 ?        00:02:52 /opt/zenoss/bin/python /opt/zenoss/Products/ZenEvents/zensyslog.py --configfile /opt/zenoss/etc/zensyslog.conf --cycle --daemon --useFileDescriptor=4

    [zenoss@zen42 4.2]$

     

    The answer is zensyslog - oh frabjous day!

     

    2) The zensyslog daemon is fairly unique in that it can log the incoming syslog messages - change the logorig parameter from False to True and restart zensyslog.  I use this to prove that syslog events are actually arriving at Zenoss.  If there is nothing there, the problem is with the configuration in your syslog servers or it is a comms problem (like firewalls not permitting UDP/514).

     

    3) If you are getting syslog message into the logorig file then increase the logging on zensyslog.  You can either do this from the GUI or by hacking the file directly (as with all daemon config files).  You need to set logseverity to 10 (Debug if you do it through the GUI) and then restart zensyslog. 

     

    What is generating your original syslog messages?  The zensyslog code can recognise most variants of syslog files but it may be that you have something that it fails to recognise.

     

    You may find my paper, "Event Management for Zenoss Core 4" helpful - http://www.skills-1st.co.uk/papers/jane/zenoss4-events/ . Chapter 4 is on syslog processing. 

     

    I have also just announced dates for the 3-day Zenoss Event Management workshop on May 29-31 and June 11-13 - see  message/72840#72840   and http://www.skills-1st.co.uk/products/courses/zenoss-events.html .

     

    Cheers,

    Jane

  • heinds Rank: White Belt
    heinds
    16 posts since
    Apr 10, 2013
    Currently Being Moderated
    3. Apr 15, 2013 9:53 AM (in response to jcurry)
    Re: ZenSyslog Question-Please assist

    Hi Jane,

     

    Thank you so much for your reply I really appreciate it.

     

    Ok, baby steps. This is what I got, do you see anything wrong?

     

     

    [root@localhost ~]# lsof -i -P | grep 514

    python     3159    zenoss   12u  IPv4  13481      0t0  UDP *:514

    [root@localhost ~]# ps -ef |grep 13481

    root     10594  4129  0 13:56 pts/0    00:00:00 grep 13481

     

    Thank you!

  • jcurry ZenossMaster
    jcurry
    1,021 posts since
    Apr 15, 2008
    Currently Being Moderated
    4. Apr 15, 2013 11:00 AM (in response to heinds)
    Re: ZenSyslog Question-Please assist

    Your second command you need to grep for 3159 as that is the process id of the python process. It is the second field that gives you the process id; the 13481 is apparently the "Device" field for lsof - not quite sure what this is.

     

    The problem with lsof as that it only gives the initial command (python) and all the zenoss daemon process lines start with python. 

     

    Cheers,

    Jane

  • heinds Rank: White Belt
    heinds
    16 posts since
    Apr 10, 2013
    Currently Being Moderated
    5. Apr 15, 2013 12:11 PM (in response to jcurry)
    Re: ZenSyslog Question-Please assist

    Hi Jane,

     

    Apologies, here is the info. If I followed your first instructions everything seems fine?

     

    [root@localhost ~]# ps -ef |grep 3159

    zenoss    3159     1  0 08:57 ?        00:00:03 /opt/zenoss/bin/python /opt/zenoss/Products/ZenEvents/zensyslog.py --configfile /opt/zenoss/etc/zensyslog.conf --cycle --daemon --useFileDescriptor=4

    root     15385  4129  0 17:54 pts/0    00:00:00 grep 3159

     

    I also changed the logorig parameter from False to True and restarted zensyslog.

     

    Here is what I got:

     

    2013-04-15 18:00:07,032 INFO zen.zensyslog: Deleting PID file /opt/zenoss/var/zensyslog-localhost.pid ...

    2013-04-15 18:00:07,032 INFO zen.zensyslog: Daemon SyslogDaemon shutting down

    2013-04-15 18:00:07,043 ERROR zen.collector.config: Task configLoader configure failed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.

    ]

    2013-04-15 18:00:10,611 INFO zen.zensyslog: Connecting to localhost:8789

    2013-04-15 18:00:10,614 INFO zen.zensyslog: Connected to ZenHub

    2013-04-15 18:00:10,617 INFO zen.maintenance: Performing periodic maintenance

    2013-04-15 18:00:10,617 INFO zen.zensyslog: Counter eventCount, value 21

    2013-04-15 18:00:10,618 INFO zen.zensyslog: 0 devices processed (0 datapoints)

    2013-04-15 18:00:10,618 INFO zen.collector.scheduler: Tasks: 1 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

    2013-04-15 18:00:15,672 INFO zen.zensyslog: Connecting to localhost:8789

    2013-04-15 18:00:15,678 INFO zen.zensyslog: Connected to ZenHub

    2013-04-15 18:00:15,681 INFO zen.maintenance: Performing periodic maintenance

    2013-04-15 18:00:15,681 INFO zen.zensyslog: Counter eventCount, value 22

    2013-04-15 18:00:15,682 INFO zen.zensyslog: 0 devices processed (0 datapoints)

    2013-04-15 18:00:15,682 INFO zen.collector.scheduler: Tasks: 1 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

    2013-04-15 18:05:15,692 INFO zen.maintenance: Performing periodic maintenance

    2013-04-15 18:05:15,693 INFO zen.zensyslog: Counter eventCount, value 22

    2013-04-15 18:05:15,702 INFO zen.zensyslog: 1 devices processed (0 datapoints)

    2013-04-15 18:05:15,702 INFO zen.collector.scheduler: Tasks: 3 Successful_Runs: 1 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

     

    I configured a ciscto 3750 switch to forward syslog messages to the zenoss server IP and I have done the same with a Zhone server.

     

    Regards

    Hein

  • jcurry ZenossMaster
    jcurry
    1,021 posts since
    Apr 15, 2008
    Currently Being Moderated
    6. Apr 15, 2013 1:04 PM (in response to heinds)
    Re: ZenSyslog Question-Please assist

    So you should now have a file called origsyslog.log in $ZENHOME/log, in addition to your zensyslog.log??  Does that have any incoming syslog messages in it?

     

    Cheers,

    Jane

  • heinds Rank: White Belt
    heinds
    16 posts since
    Apr 10, 2013
    Currently Being Moderated
    7. Apr 16, 2013 2:56 AM (in response to jcurry)
    Re: ZenSyslog Question-Please assist

    Hi Jane,

     

    I located the file in /opt/zenoss/log/origsyslog.log and opened it in windows but there is nothing in it.

     

    PS: Not sure how to open it in centos as you can probably gather.... ;(

     

    What does it mean if it is empty, no logs coming through?

  • jcurry ZenossMaster
    jcurry
    1,021 posts since
    Apr 15, 2008
    Currently Being Moderated
    8. Apr 16, 2013 4:09 AM (in response to heinds)
    Re: ZenSyslog Question-Please assist

    Hmm - sounds like there are no syslog events arriving.  If you change directory to $ZENHOME/log and do a ls -l on the file,  when was it last updated?  Might it have been  when you changed the logorig parameter?  For example:

     

    [zenoss@zen42 log]$ cd /opt/zenoss/log/

    [zenoss@zen42 log]$ ls -l origsyslog.log

    -rwxrwxr-x 1 zenoss zenoss 28613171 Apr 16 09:00 origsyslog.log

    [zenoss@zen42 log]$ less origsyslog.log

    [zenoss@zen42 log]$

     

    The date is the last time the file was updated. the big number (28613171) is the size (which should obviously increase whenever a syslog message is received).  less is a unix command to display the contents of a file (space gets the next page, carriage-return gets the next line, if you know some vi editor commands then they work too).

     

    If this file is virtually emptry and not updating when you believe it should be recieving events, then I would certainly focus attention on the syslog systems that are supposed to be forwarding events to Zenoss and at any fiewall that might be in between.

     

    One test I usually start with is to configure the Zenoss server so that his own syslog events are forwarded to Zenoss (have a look at the Event Management paper for an example).  If you can get those flowing then you know that zensyslog is capable of reading and interpreting events. and you can focus your attention on the delivery mechanism.

     

    Cheers,

    Jane

  • heinds Rank: White Belt
    heinds
    16 posts since
    Apr 10, 2013
    Currently Being Moderated
    9. Apr 16, 2013 10:52 AM (in response to jcurry)
    Re: ZenSyslog Question-Please assist

    Hi Jane,

     

    Thank you for your assistance thus far and for not giving up on me!

     

    I ran the following test to verify that the cisco is actually sending out syslog messages.

     

    I downloaded Tftpd32, it has a very simple built in syslog server that you can use. I installed it, ran wireshark and changed the cisco syslog destination from the zenoss server to my laptop interface IP.

     

    I immedietly received syslog messages from the cisco and wireshark confirmed syslog messages sent from the cisco to my laptop!

     

    I logged back into the cisco and changed it back to the zenoss ip address.

    Neither Zenoss or wireshark picks up anything...

     

     

    My laptop also connects to the same switch as the zenoss server with no firewall enabled.

     

    This just keeps getting weirder and weirder... Any more tests to run?

     

    I did what you reccommended and got this:

     

    [root@localhost log]# ls -l origsyslog.log

    -rw-r--r-- 1 zenoss zenoss 0 Apr 15 18:00 origsyslog.log

     

    (Not looking good)

     

    And so summerise:

     

    [root@localhost log]# lsof -i -P | grep 514

    python    14524    zenoss   49u  IPv4 938605      0t0  UDP *:514

     

    [root@localhost log]# ps -ef |grep 938605

    root     17724 21799  0 16:49 pts/0    00:00:00 grep 938605

     

    [root@localhost log]# ps -ef |grep 14524

    zenoss   14524     1  0 14:24 ?        00:00:01 /opt/zenoss/bin/python /opt/zenoss/Products/ZenEvents/zensyslog.py --configfile /opt/zenoss/etc/zensyslog.conf --cycle --daemon --useFileDescriptor=15

    root     17741 21799  0 16:50 pts/0    00:00:00 grep 14524

  • jcurry ZenossMaster
    jcurry
    1,021 posts since
    Apr 15, 2008
    Currently Being Moderated
    10. Apr 16, 2013 11:17 AM (in response to heinds)
    Re: ZenSyslog Question-Please assist

    Good one isn't it?

     

    Are you running wireshark on your laptop?

     

    Is there a firewall running on your Zenoss server?  As root, run:

    iptables --list -v

     

    I have a line like:

    0 0 ACCEPT udp  --  anyany anywhere         anywhere        state NEW udp dpt:syslog

     

    The Zenoss server's firewall would block stuff before it gets to zensyslog.

     

    Cheers,

    Jane

  • heinds Rank: White Belt
    heinds
    16 posts since
    Apr 10, 2013
    Currently Being Moderated
    11. Apr 16, 2013 1:40 PM (in response to jcurry)
    Re: ZenSyslog Question-Please assist

    Hey Jane,

     

    Here is the output, not seeying any thing on UDP...

     

    [root@localhost log]# iptables --list -v

    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)

    pkts bytes target     prot opt in     out     source               destination        

    3250K 1163M ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED

        0     0 ACCEPT     icmp --  any    any     anywhere             anywhere           

    1087 63796 ACCEPT     all  --  lo     any     anywhere             anywhere           

        0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere            state NEW tcp dpt:ssh

    19730 2364K REJECT     all  --  any    any     anywhere             anywhere            reject-with icmp-host-prohibited

     

    Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

    pkts bytes target     prot opt in     out     source               destination        

        0     0 REJECT     all  --  any    any     anywhere             anywhere            reject-with icmp-host-prohibited

     

    Chain OUTPUT (policy ACCEPT 3253K packets, 1044M bytes)

    pkts bytes target     prot opt in     out     source               destination

  • jcurry ZenossMaster
    jcurry
    1,021 posts since
    Apr 15, 2008
    Currently Being Moderated
    12. Apr 16, 2013 6:34 PM (in response to heinds)
    Re: ZenSyslog Question-Please assist

    I think that means that your firewall on your Zenoss server is rejecting incoming syslog messages on udp/514.

     

    Use the System -> Administration -> firewall menus.  Under "Other Ports" you may or may not have anything.  You need to ensure that UDP/514 is selected here; using the Add button gives you a scrolled list and you should be able to select what you want.  I attach a screenshot of my configs.

    centos_firewall_other_ports.jpg

    If you ever want your Zenoss to receive SNMP TRAPs, also add the two entries for TCP and UDP 162.

     

    I believe the first line of your current firewall config is permitting traffic in that is in response to a message from the Zenoss server, so things like SNMP get/set will be allowed in as the conversation starts from Zenoss but both syslog and SNMP TRAPs are messages unsolicited by Zenoss - hence the conversation starts from the outside and you have no firewall rules that allow these message types in.

     

    Cheers,

    Jane

  • heinds Rank: White Belt
    heinds
    16 posts since
    Apr 10, 2013
    Currently Being Moderated
    13. Apr 20, 2013 10:47 AM (in response to jcurry)
    Re: ZenSyslog Question-Please assist

    Hi Jane,

     

    I just wanted to let you know that......it worked!!!

     

    Thank you so much for your patience with my beginner questions, you should concider becoming a teacher. If you didnt come to the rescue I would never had figured out the problem!

     

    Next step is to sort out the events and how Zenoss displays it but just receiving emails is good for now!

     

    Again, thank you so much!

  • guyverix ZenossMaster
    guyverix
    846 posts since
    Jul 10, 2007
    Currently Being Moderated
    14. Apr 20, 2013 2:09 PM (in response to heinds)
    Re: ZenSyslog Question-Please assist

    Grin, I think Jane is a teacher.. 

     

    Dont forget to mark the question as answered

1 2 Previous Next

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points