Archived community.zenoss.org | full text search
Skip navigation
Up to Discussions in zenoss-users
19627 Views 5 Replies Latest reply: Oct 13, 2008 3:15 PM by beanfield RSS
beanfield Rank: Green Belt
beanfield
161 posts since
Apr 16, 2008
Currently Being Moderated

Oct 9, 2008 12:38 PM

threshold of zenperfsnmp cycle time exceeded

In the event console, I regularly receive the following critical alert:

threshold of zenperfsnmp cycle time exceeded: current value 243.93


I had a couple questions regarding why I'm getting this alert. Firstly, if I go to /Monitors/localhost (my only collector) and view the Performance tab, the "Cycle Times" values for zenperfsnmp for the past 36 hours are as follows:
cur 82.07
avg: 73.35
max: 127.56

So if the graphs are showing cycle times far less than 243.93...then why am I receiving this alert?

Also if I go to /Monitors/localhost and view the Edit tab, my "SNMP Performance Cycle Interval (secs)" is set to 300. So if I'm not exceeding this value...why am I being alerted about it?

Also, it seems very strange that the "current value" is always the same number until a restart....as the count on that alert keeps increasing. If I go through my history, I can see the same alert with slightly different values....but the times they change seem to coincide with a restart of zenoss. For instance, I have on of these alerts that has a count of 173...and another that has a count of 143. I find it hard to believe that zenperfsnmp took 244.57 seconds EXACTLY 173 times in a row.
  • beanfield Rank: Green Belt
    beanfield
    161 posts since
    Apr 16, 2008
    Currently Being Moderated
    1. Oct 9, 2008 12:48 PM (in response to beanfield)
    RE: threshold of zenperfsnmp cycle time exceeded
    Also, running zenperfsnmp in the foreground I can see it dies collecting from a different device each time:

    INFO:zen.zenperfsnmp:Daemon zenperfsnmp shutting down
WARNING:zen.zenperfsnmp:Error loading config for devices ['linuscs34.gsc.wustl.edu', 'vm2.gsc.wustl.edu']
DEBUG:zen.zenperfsnmp:Fetching config for ['linuscs39.gsc.wustl.edu', 'linuscs40.gsc.wustl.edu']
ERROR:zen.zenperfsnmp:Error loading devices: [Failure instance: Traceback (failure with no frames): twisted.spread.pb.DeadReferenceError: Calling Stale Broker
]
DEBUG:zen.zenperfsnmp:removing service EventService
DEBUG:zen.zenperfsnmp:removing service SnmpPerfConfig
INFO:zen.zenperfsnmp:zenperfsnmp shutting down


    INFO:zen.zenperfsnmp:Daemon zenperfsnmp shutting down
WARNING:zen.zenperfsnmp:Error loading config for devices ['linuscs34.gsc.wustl.edu', 'vm2.gsc.wustl.edu']
DEBUG:zen.zenperfsnmp:Fetching config for ['linuscs39.gsc.wustl.edu', 'linuscs40.gsc.wustl.edu']
ERROR:zen.zenperfsnmp:Error loading devices: [Failure instance: Traceback (failure with no frames): twisted.spread.pb.DeadReferenceError: Calling Stale Broker
]
DEBUG:zen.zenperfsnmp:removing service EventService
DEBUG:zen.zenperfsnmp:removing service SnmpPerfConfig
INFO:zen.zenperfsnmp:zenperfsnmp shutting down

    
WARNING:zen.zenperfsnmp:Error loading config for devices ['linuscs56.gsc.wustl.edu', 'linuscs33.gsc.wustl.edu']
DEBUG:zen.zenperfsnmp:Fetching config for ['linuscs47.gsc.wustl.edu', 'linuscs48.gsc.wustl.edu']
ERROR:zen.zenperfsnmp:Error loading devices: [Failure instance: Traceback (failure with no frames): twisted.spread.pb.DeadReferenceError: Calling Stale Broker
]
DEBUG:zen.zenperfsnmp:removing service EventService
DEBUG:zen.zenperfsnmp:removing service SnmpPerfConfig
INFO:zen.zenperfsnmp:zenperfsnmp shutting down


    I'm beginning to think it's related to this past issue. http://forums.zenoss.com/viewtopic.php?t=7277&highlight=localhost

    No errors are being shown in zenhub when running in the foreground. Additionally, there are no pickle files for any of the servers it's failing on.
  • beanfield Rank: Green Belt
    beanfield
    161 posts since
    Apr 16, 2008
    Currently Being Moderated
    2. Oct 9, 2008 2:23 PM (in response to beanfield)
    RE: threshold of zenperfsnmp cycle time exceeded
    I followed some of the tips from this thread http://forums.zenoss.com/viewtopic.php?t=5304&postdays=0&postorder=asc&highlight=twisted+spread++deadreferenceerror&start=30

    Someone mentioned running a "zenmigrate". I ran this and it appears that corrected the issue with zenperfsnmp bombing out. I can only assume I neglected to run this on my last upgrade.

    I also ran the zendmd script posted by cluther, which cleared out some other issues with stale/moved devices.

    Now I get an "error" alert (instead of critical) for zenperfsnmp taking too long, which is an improvement. 

    threshold of zenperfsnmp cycle time exceeded: current value 245.28


    I can only assume it was critical before because it bombed out. However, my zenperfsnmp cycle time graphs still show an average of around 70 with a max of 127. I still don't understand the discrepancy between the time reflected in the alert and the time being shown in the graph.
  • beanfield Rank: Green Belt
    beanfield
    161 posts since
    Apr 16, 2008
    Currently Being Moderated
    3. Oct 9, 2008 4:07 PM (in response to beanfield)
    RE: threshold of zenperfsnmp cycle time exceeded
    Nevermind...now it's back to "critical" with a timeout of 244.16. Why it was less severe (error) when it took longer (245.28 ) is beyond me.
  • artifact Rank: Green Belt
    artifact
    303 posts since
    Mar 11, 2008
    Currently Being Moderated
    4. Oct 10, 2008 1:31 PM (in response to beanfield)
    RE: threshold of zenperfsnmp cycle time exceeded
    What is your zenperfsnmp datapoint rate?

    I have ~260 datapoint rate with a max cycle time of 60...

    So my guess would be that its one particular device thats giving zenperfsnmp trouble. I've seen this happen with my BGP routers: zenperfsnmp was trying to pull the entire routing table off of them, hence the timeouts... so I just turned off the RouteMap collector plugin on those devices, and that fixed it for me.
  • beanfield Rank: Green Belt
    beanfield
    161 posts since
    Apr 16, 2008
    Currently Being Moderated
    5. Oct 13, 2008 3:15 PM (in response to artifact)
    RE: threshold of zenperfsnmp cycle time exceeded

    "webpass" wrote:

     

    What is your zenperfsnmp datapoint rate?

    I have ~260 datapoint rate with a max cycle time of 60...

    So my guess would be that its one particular device thats giving zenperfsnmp trouble. I've seen this happen with my BGP routers: zenperfsnmp was trying to pull the entire routing table off of them, hence the timeouts... so I just turned off the RouteMap collector plugin on those devices, and that fixed it for me.



    I'm not sure if I follow for the zenperfsnmp data point rate and max cycle time. Under /Monitors/localhost "SNMP Performance Cycle Interval (secs)" is set to 300. All of my devices have the zProperty zSnmpTimeout set to the default 2.5.

More Like This

  • Retrieving data ...