Archived community.zenoss.org | full text search
Skip navigation
1 2 Previous Next 178096 Views 18 Replies Latest reply: Nov 3, 2009 5:06 PM by Captain MESO RSS
Kamil Kisiel Rank: White Belt 46 posts since
Feb 12, 2009
Currently Being Moderated

Oct 28, 2009 6:37 PM

zenstatus issues after 2.4.5 -> 2.5 upgrade

I upgraded my stack install of 2.4.5 to 2.5 today and after fixing a few other minor problems I still have one remaining. Both my primary and secondary collector show zenstatus as having issues, and the log is full of messages like the following:

 

2009-10-28 15:30:53,394 WARNING zen.ZenStatus: Scan cycle not complete in 119.99 seconds

2009-10-28 15:30:53,394 WARNING zen.ZenStatus: Waiting for the cycle to complete

2009-10-28 15:31:53,398 WARNING zen.ZenStatus: Scan cycle not complete in 179.99 seconds

2009-10-28 15:31:53,398 WARNING zen.ZenStatus: Restarting jobs for another cycle

2009-10-28 15:32:53,402 WARNING zen.ZenStatus: Scan cycle not complete in 59.98 seconds

2009-10-28 15:32:53,402 WARNING zen.ZenStatus: Waiting for the cycle to complete

2009-10-28 15:33:53,402 WARNING zen.ZenStatus: Scan cycle not complete in 119.98 seconds

2009-10-28 15:33:53,402 WARNING zen.ZenStatus: Waiting for the cycle to complete

2009-10-28 15:34:53,406 WARNING zen.ZenStatus: Scan cycle not complete in 179.99 seconds

2009-10-28 15:34:53,406 WARNING zen.ZenStatus: Restarting jobs for another cycle

2009-10-28 15:35:53,410 WARNING zen.ZenStatus: Scan cycle not complete in 59.99 seconds

2009-10-28 15:35:53,410 WARNING zen.ZenStatus: Waiting for the cycle to complete

Does anyone have an idea of what's going on here?
  • Captain MESO Rank: Green Belt 114 posts since
    Mar 5, 2008
    Currently Being Moderated
    1. Oct 28, 2009 7:16 PM (in response to Kamil Kisiel)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

    I don't have a solution but can confirm this problem. I have exactly the same issue since upgrading from 2.4.5 to 2.5. Any hints?

     

    2009-10-28 21:41:42,548 INFO zen.ZenStatus: Connected to ZenHub
    2009-10-28 21:41:42,582 INFO zen.ZenStatus: Fetching property items
    2009-10-28 21:42:42,311 INFO zen.ZenStatus: Fetching default RRDCreateCommand
    2009-10-28 21:42:43,550 INFO zen.ZenStatus: Getting threshold classes
    2009-10-28 21:42:45,713 INFO zen.ZenStatus: Getting collector thresholds
    2009-10-28 21:45:31,860 WARNING zen.ZenStatus: Scan cycle not complete in 32.23 seconds
    2009-10-28 21:45:31,868 WARNING zen.ZenStatus: Waiting for the cycle to complete
    2009-10-28 21:47:31,901 WARNING zen.ZenStatus: Scan cycle not complete in 152.27 seconds
    2009-10-28 21:47:31,904 WARNING zen.ZenStatus: Waiting for the cycle to complete
    2009-10-28 21:49:31,965 WARNING zen.ZenStatus: Scan cycle not complete in 272.34 seconds
    2009-10-28 21:49:31,967 WARNING zen.ZenStatus: Restarting jobs for another cycle
    2009-10-28 21:51:31,977 WARNING zen.ZenStatus: Scan cycle not complete in 79.62 seconds
    2009-10-28 21:51:31,980 WARNING zen.ZenStatus: Waiting for the cycle to complete
    2009-10-28 21:53:31,982 WARNING zen.ZenStatus: Scan cycle not complete in 199.63 seconds
    2009-10-28 21:53:31,982 WARNING zen.ZenStatus: Waiting for the cycle to complete
    2009-10-28 21:55:32,081 WARNING zen.ZenStatus: Scan cycle not complete in 319.73 seconds
    2009-10-28 21:55:32,086 WARNING zen.ZenStatus: Restarting jobs for another cycle
    2009-10-28 21:57:32,085 WARNING zen.ZenStatus: Scan cycle not complete in 112.45 seconds
    2009-10-28 21:57:32,086 WARNING zen.ZenStatus: Waiting for the cycle to complete
    
  • guyverix ZenossMaster 846 posts since
    Jul 10, 2007
    Currently Being Moderated
    2. Oct 28, 2009 7:24 PM (in response to Captain MESO)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade
    Any chance you could turn on verbose logging for zenstatus and zenhub in the configs?
  • Captain MESO Rank: Green Belt 114 posts since
    Mar 5, 2008
    Currently Being Moderated
    3. Oct 29, 2009 1:00 AM (in response to guyverix)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

    Thanks guyverix. Verbose logs for zenstatus and zenhub seems to be ok for me. But zenaction logging is very interesting. Since upgrading it shows:

     

    2009-10-28 21:24:09,744 INFO zen.ZenActions: Processed 0 commands in 0.005499 2009-10-28 21:24:10,291 ERROR zen.ZenActions: unexpected exception Traceback (most recent call last):   File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle     self.mainbody()   File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 557, in mainbody     self.heartbeatEvents()   File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 488, in heartbeatEvents     hostname = self.fetchMonitorHostname(monitor)   File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 463, in fetchMonitorHostname     hostname = self.dmd.Monitors.getPerformanceMonitor(monitor).hostname

    AttributeError: hostname
    

     

     


    I think this is causing the zenstatus problem. What could it be?

  • Captain MESO Rank: Green Belt 114 posts since
    Mar 5, 2008
    Currently Being Moderated
    4. Oct 29, 2009 1:16 AM (in response to Captain MESO)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

    Verbose log for zenaction:

     

    2009-10-29 06:13:56,610 INFO zen.ZenActions: Processed 0 commands in 0.000154
    2009-10-29 06:13:56,611 DEBUG zen.ZenActions: call age_events(96, 5);
    2009-10-29 06:13:56,612 DEBUG zen.DbConnectionPool: Retrieved a connection; Pool size: 0
    2009-10-29 06:13:56,617 DEBUG zen.DbConnectionPool: Returned a connection; Pool size: 1
    2009-10-29 06:13:56,618 DEBUG zen.ZenActions: SELECT monitor, component FROM status WHERE eventClass = '/Status/Heartbeat'
    2009-10-29 06:13:56,618 DEBUG zen.DbConnectionPool: Retrieved a connection; Pool size: 0
    2009-10-29 06:13:56,630 DEBUG zen.DbConnectionPool: Returned a connection; Pool size: 1
    2009-10-29 06:13:56,630 DEBUG zen.ZenActions: SELECT device, component FROM heartbeat WHERE DATE_ADD(lastTime, INTERVAL timeout SECOND) <= NOW();
    2009-10-29 06:13:56,630 DEBUG zen.DbConnectionPool: Retrieved a connection; Pool size: 0
    2009-10-29 06:13:56,638 DEBUG zen.DbConnectionPool: Returned a connection; Pool size: 1
    2009-10-29 06:13:56,639 ERROR zen.ZenActions: unexpected exception
    Traceback (most recent call last):
      File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle
        self.mainbody()
      File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 557, in mainbody
        self.heartbeatEvents()
      File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 488, in heartbeatEvents
        hostname = self.fetchMonitorHostname(monitor)
      File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 463, in fetchMonitorHostname
        hostname = self.dmd.Monitors.getPerformanceMonitor(monitor).hostname
    AttributeError: hostname
    
  • mdereus Newbie 2 posts since
    Oct 20, 2009
    Currently Being Moderated
    5. Oct 29, 2009 10:39 AM (in response to Kamil Kisiel)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

    I too am getting the same errors in zenstatus and zenactions which have occurred from the moment of upgrading from 2.4.5 to 2.5.

     

     


  • Captain MESO Rank: Green Belt 114 posts since
    Mar 5, 2008
    Currently Being Moderated
    7. Oct 30, 2009 2:24 AM (in response to Kamil Kisiel)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

    Thanks a lot kisielk. 5714 fixed zenactions.

     

    Zenstatus problem still remains after 2.4.5 --> 2.5 updgrade. Any solution available for this problem?

  • geotek Rank: White Belt 81 posts since
    Jun 11, 2007
    Currently Being Moderated
    9. Oct 30, 2009 4:35 AM (in response to Kamil Kisiel)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

    I am also seeing the issue that zenstatus.log is crashing every minute. This occurs on the 2.5 VMware image...

     


    2009-10-30 07:34:33,535 ERROR zen.ZenActions: unexpected exception
    Traceback (most recent call last):
      File "/home/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle
        self.mainbody()
      File "/home/zenoss/Products/ZenEvents/zenactions.py", line 557, in mainbody
        self.heartbeatEvents()
      File "/home/zenoss/Products/ZenEvents/zenactions.py", line 488, in heartbeatEvents
        hostname = self.fetchMonitorHostname(monitor)
      File "/home/zenoss/Products/ZenEvents/zenactions.py", line 463, in fetchMonitorHostname
        hostname = self.dmd.Monitors.getPerformanceMonitor(monitor).hostname
    AttributeError: hostname

     

    as well as on our upgraded SUSE 10.2 production machine::


    2009-10-30 08:34:01,683 ERROR zen.ZenActions: unexpected exception
    Traceback (most recent call last):
      File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle
        self.mainbody()
      File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 552, in mainbody
        self.eventCommands(zem)
      File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 541, in eventCommands
        self.processEvent(zem, command, self.runEventCommand)
      File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 258, in processEvent
        fields = context.getEventFields()
      File "/usr/local/zenoss/Products/ZenEvents/EventCommand.py", line 63, in getEventFields
        return self.eventManager.getFieldList()
    AttributeError: eventManager

     

    Replacing the file zenactions.py from changeset 15582 didn't help, the zenactions daemon won't even start with it (no error log entry)

     

    Any ideas?

     

    Martin

     

    dorfanger-blankenburg.de

  • Chet Luther ZenossEmployee 1,302 posts since
    May 22, 2007
    Currently Being Moderated
    10. Oct 30, 2009 3:03 PM (in response to Kamil Kisiel)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

    I've seen this happen before with zenmodeler. The issue ended up being that a value that shouldn't be stored to the object database did get stored. The fix in that case was to run "zendmd" as the zenoss user and paste in the following snippet.

     

    del(dmd.Monitors.Performance.localhost._properties)

    commit()

  • geotek Rank: White Belt 81 posts since
    Jun 11, 2007
    Currently Being Moderated
    11. Oct 30, 2009 3:54 PM (in response to Chet Luther)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

    Chet,

     

    that command doesn't seem to work on either of the two machines:

     

    >>> del(dmd.Monitors.Performance.localhost._properties)
    Traceback (most recent call last):
      File "<console>", line 1, in ?
    AttributeError: _properties

     


    However, on my upgraded machine I found the following left over entry from helloWorld Zenpack:

     

    /zport/dmd/ZenEventManager/commands/helloWorld

     

    Could this be the culprit on my upgraded machine? If it is, how can I get rid of this dummy helloWorld object? (d_delObject('helloWorld') fails with "AttributeError: eventManager" and installation and removal of the old Zenpack doesn' t delete it either)

     

    Martin

     

    dorfanger-blankenburg.de

     

     


  • Captain MESO Rank: Green Belt 114 posts since
    Mar 5, 2008
    Currently Being Moderated
    12. Oct 30, 2009 7:02 PM (in response to Chet Luther)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

    Thanks Chet, but I get the same error message:

     

    >>> del(dmd.Monitors.Performance.localhost._properties)
    Traceback (most recent call last):
      File "<console>", line 1, in ?
    AttributeError: _properties

    Typo?

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009
    Currently Being Moderated
    13. Nov 2, 2009 1:57 PM (in response to Captain MESO)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade
    Just out of curiosity, did you guys have the WMI Data Source ZenPack installed when you tried to upgrade to 2.5?
  • mdereus Newbie 2 posts since
    Oct 20, 2009
    Currently Being Moderated
    14. Nov 2, 2009 3:37 PM (in response to Ryan Matte)
    Re: zenstatus issues after 2.4.5 -> 2.5 upgrade
    Yes, I did and still do.
1 2 Previous Next

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points