zenstatus issues after 2.4.5 -> 2.5 upgrade

Up to Discussions in zenoss-users

1 2 Previous Next 178092 Views 18 Replies Latest reply: Nov 3, 2009 5:06 PM by Captain MESO

Kamil Kisiel

46 posts since
Feb 12, 2009

Currently Being Moderated

Oct 28, 2009 6:37 PM

I upgraded my stack install of 2.4.5 to 2.5 today and after fixing a few other minor problems I still have one remaining. Both my primary and secondary collector show zenstatus as having issues, and the log is full of messages like the following:

2009-10-28 15:30:53,394 WARNING zen.ZenStatus: Scan cycle not complete in 119.99 seconds

2009-10-28 15:30:53,394 WARNING zen.ZenStatus: Waiting for the cycle to complete

2009-10-28 15:31:53,398 WARNING zen.ZenStatus: Scan cycle not complete in 179.99 seconds

2009-10-28 15:31:53,398 WARNING zen.ZenStatus: Restarting jobs for another cycle

2009-10-28 15:32:53,402 WARNING zen.ZenStatus: Scan cycle not complete in 59.98 seconds

2009-10-28 15:32:53,402 WARNING zen.ZenStatus: Waiting for the cycle to complete

2009-10-28 15:33:53,402 WARNING zen.ZenStatus: Scan cycle not complete in 119.98 seconds

2009-10-28 15:33:53,402 WARNING zen.ZenStatus: Waiting for the cycle to complete

2009-10-28 15:34:53,406 WARNING zen.ZenStatus: Scan cycle not complete in 179.99 seconds

2009-10-28 15:34:53,406 WARNING zen.ZenStatus: Restarting jobs for another cycle

2009-10-28 15:35:53,410 WARNING zen.ZenStatus: Scan cycle not complete in 59.99 seconds

2009-10-28 15:35:53,410 WARNING zen.ZenStatus: Waiting for the cycle to complete

Does anyone have an idea of what's going on here?

Like (0)

Captain MESO

114 posts since
Mar 5, 2008

Currently Being Moderated

1. Oct 28, 2009 7:16 PM (in response to Kamil Kisiel)

Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

I don't have a solution but can confirm this problem. I have exactly the same issue since upgrading from 2.4.5 to 2.5. Any hints?

2009-10-28 21:41:42,548 INFO zen.ZenStatus: Connected to ZenHub
2009-10-28 21:41:42,582 INFO zen.ZenStatus: Fetching property items
2009-10-28 21:42:42,311 INFO zen.ZenStatus: Fetching default RRDCreateCommand
2009-10-28 21:42:43,550 INFO zen.ZenStatus: Getting threshold classes
2009-10-28 21:42:45,713 INFO zen.ZenStatus: Getting collector thresholds
2009-10-28 21:45:31,860 WARNING zen.ZenStatus: Scan cycle not complete in 32.23 seconds
2009-10-28 21:45:31,868 WARNING zen.ZenStatus: Waiting for the cycle to complete
2009-10-28 21:47:31,901 WARNING zen.ZenStatus: Scan cycle not complete in 152.27 seconds
2009-10-28 21:47:31,904 WARNING zen.ZenStatus: Waiting for the cycle to complete
2009-10-28 21:49:31,965 WARNING zen.ZenStatus: Scan cycle not complete in 272.34 seconds
2009-10-28 21:49:31,967 WARNING zen.ZenStatus: Restarting jobs for another cycle
2009-10-28 21:51:31,977 WARNING zen.ZenStatus: Scan cycle not complete in 79.62 seconds
2009-10-28 21:51:31,980 WARNING zen.ZenStatus: Waiting for the cycle to complete
2009-10-28 21:53:31,982 WARNING zen.ZenStatus: Scan cycle not complete in 199.63 seconds
2009-10-28 21:53:31,982 WARNING zen.ZenStatus: Waiting for the cycle to complete
2009-10-28 21:55:32,081 WARNING zen.ZenStatus: Scan cycle not complete in 319.73 seconds
2009-10-28 21:55:32,086 WARNING zen.ZenStatus: Restarting jobs for another cycle
2009-10-28 21:57:32,085 WARNING zen.ZenStatus: Scan cycle not complete in 112.45 seconds
2009-10-28 21:57:32,086 WARNING zen.ZenStatus: Waiting for the cycle to complete

Report Abuse
Like (0)

guyverix
846 posts since
Jul 10, 2007

Currently Being Moderated

2. Oct 28, 2009 7:24 PM (in response to Captain MESO)
Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

Any chance you could turn on verbose logging for zenstatus and zenhub in the configs?

Report Abuse

Like (0)

Captain MESO

114 posts since
Mar 5, 2008

Currently Being Moderated

3. Oct 29, 2009 1:00 AM (in response to guyverix)

Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

Thanks guyverix. Verbose logs for zenstatus and zenhub seems to be ok for me. But zenaction logging is very interesting. Since upgrading it shows:

2009-10-28 21:24:09,744 INFO zen.ZenActions: Processed 0 commands in 0.005499 2009-10-28 21:24:10,291 ERROR zen.ZenActions: unexpected exception Traceback (most recent call last): File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle self.mainbody() File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 557, in mainbody self.heartbeatEvents() File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 488, in heartbeatEvents hostname = self.fetchMonitorHostname(monitor) File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 463, in fetchMonitorHostname hostname = self.dmd.Monitors.getPerformanceMonitor(monitor).hostname

AttributeError: hostname

I think this is causing the zenstatus problem. What could it be?

Report Abuse
Like (0)

Captain MESO

114 posts since
Mar 5, 2008

Currently Being Moderated

4. Oct 29, 2009 1:16 AM (in response to Captain MESO)

Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

Verbose log for zenaction:

2009-10-29 06:13:56,610 INFO zen.ZenActions: Processed 0 commands in 0.000154
2009-10-29 06:13:56,611 DEBUG zen.ZenActions: call age_events(96, 5);
2009-10-29 06:13:56,612 DEBUG zen.DbConnectionPool: Retrieved a connection; Pool size: 0
2009-10-29 06:13:56,617 DEBUG zen.DbConnectionPool: Returned a connection; Pool size: 1
2009-10-29 06:13:56,618 DEBUG zen.ZenActions: SELECT monitor, component FROM status WHERE eventClass = '/Status/Heartbeat'
2009-10-29 06:13:56,618 DEBUG zen.DbConnectionPool: Retrieved a connection; Pool size: 0
2009-10-29 06:13:56,630 DEBUG zen.DbConnectionPool: Returned a connection; Pool size: 1
2009-10-29 06:13:56,630 DEBUG zen.ZenActions: SELECT device, component FROM heartbeat WHERE DATE_ADD(lastTime, INTERVAL timeout SECOND) <= NOW();
2009-10-29 06:13:56,630 DEBUG zen.DbConnectionPool: Retrieved a connection; Pool size: 0
2009-10-29 06:13:56,638 DEBUG zen.DbConnectionPool: Returned a connection; Pool size: 1
2009-10-29 06:13:56,639 ERROR zen.ZenActions: unexpected exception
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle
    self.mainbody()
  File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 557, in mainbody
    self.heartbeatEvents()
  File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 488, in heartbeatEvents
    hostname = self.fetchMonitorHostname(monitor)
  File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 463, in fetchMonitorHostname
    hostname = self.dmd.Monitors.getPerformanceMonitor(monitor).hostname
AttributeError: hostname

Report Abuse
Like (0)

mdereus
2 posts since
Oct 20, 2009

Currently Being Moderated

5. Oct 29, 2009 10:39 AM (in response to Kamil Kisiel)
Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

I too am getting the same errors in zenstatus and zenactions which have occurred from the moment of upgrading from 2.4.5 to 2.5.

Report Abuse

Like (0)
Kamil Kisiel
46 posts since
Feb 12, 2009

Currently Being Moderated

6. Oct 29, 2009 12:27 PM (in response to Captain MESO)
Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

This is an unrelated problem and is covered in ticket #5714

Report Abuse

Like (0)
Captain MESO
114 posts since
Mar 5, 2008

Currently Being Moderated

7. Oct 30, 2009 2:24 AM (in response to Kamil Kisiel)
Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

Thanks a lot kisielk. 5714 fixed zenactions.

Zenstatus problem still remains after 2.4.5 --> 2.5 updgrade. Any solution available for this problem?

Report Abuse

Like (0)
Kamil Kisiel
46 posts since
Feb 12, 2009

Currently Being Moderated

8. Oct 30, 2009 2:41 AM (in response to Captain MESO)
Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

No I haven't found a solution yet, that's why I'm asking here

Report Abuse

Like (0)
geotek
81 posts since
Jun 11, 2007

Currently Being Moderated

9. Oct 30, 2009 4:35 AM (in response to Kamil Kisiel)
Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

I am also seeing the issue that zenstatus.log is crashing every minute. This occurs on the 2.5 VMware image...

2009-10-30 07:34:33,535 ERROR zen.ZenActions: unexpected exception
Traceback (most recent call last):
File "/home/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle
    self.mainbody()
File "/home/zenoss/Products/ZenEvents/zenactions.py", line 557, in mainbody
    self.heartbeatEvents()
File "/home/zenoss/Products/ZenEvents/zenactions.py", line 488, in heartbeatEvents
    hostname = self.fetchMonitorHostname(monitor)
File "/home/zenoss/Products/ZenEvents/zenactions.py", line 463, in fetchMonitorHostname
    hostname = self.dmd.Monitors.getPerformanceMonitor(monitor).hostname
AttributeError: hostname

as well as on our upgraded SUSE 10.2 production machine::

2009-10-30 08:34:01,683 ERROR zen.ZenActions: unexpected exception
Traceback (most recent call last):
File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle
    self.mainbody()
File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 552, in mainbody
    self.eventCommands(zem)
File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 541, in eventCommands
    self.processEvent(zem, command, self.runEventCommand)
File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 258, in processEvent
    fields = context.getEventFields()
File "/usr/local/zenoss/Products/ZenEvents/EventCommand.py", line 63, in getEventFields
    return self.eventManager.getFieldList()
AttributeError: eventManager

Replacing the file zenactions.py from changeset 15582 didn't help, the zenactions daemon won't even start with it (no error log entry)

Any ideas?

Martin

dorfanger-blankenburg.de

Report Abuse

Like (0)
Chet Luther
1,302 posts since
May 22, 2007

Currently Being Moderated

10. Oct 30, 2009 3:03 PM (in response to Kamil Kisiel)
Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

I've seen this happen before with zenmodeler. The issue ended up being that a value that shouldn't be stored to the object database did get stored. The fix in that case was to run "zendmd" as the zenoss user and paste in the following snippet.

del(dmd.Monitors.Performance.localhost._properties)
commit()

Report Abuse

Like (0)
geotek
81 posts since
Jun 11, 2007

Currently Being Moderated

11. Oct 30, 2009 3:54 PM (in response to Chet Luther)
Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

Chet,

that command doesn't seem to work on either of the two machines:

>>> del(dmd.Monitors.Performance.localhost._properties)
Traceback (most recent call last):
File "<console>", line 1, in ?
AttributeError: _properties

However, on my upgraded machine I found the following left over entry from helloWorld Zenpack:

/zport/dmd/ZenEventManager/commands/helloWorld

Could this be the culprit on my upgraded machine? If it is, how can I get rid of this dummy helloWorld object? (d_delObject('helloWorld') fails with "AttributeError: eventManager" and installation and removal of the old Zenpack doesn' t delete it either)

Martin

dorfanger-blankenburg.de

Report Abuse

Like (0)
Captain MESO
114 posts since
Mar 5, 2008

Currently Being Moderated

12. Oct 30, 2009 7:02 PM (in response to Chet Luther)
Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

Thanks Chet, but I get the same error message:

>>> del(dmd.Monitors.Performance.localhost._properties)
Traceback (most recent call last):
File "<console>", line 1, in ?
AttributeError: _properties
Typo?

Report Abuse

Like (0)
Ryan Matte
653 posts since
Mar 26, 2009

Currently Being Moderated

13. Nov 2, 2009 1:57 PM (in response to Captain MESO)
Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

Just out of curiosity, did you guys have the WMI Data Source ZenPack installed when you tried to upgrade to 2.5?

Report Abuse

Like (0)
mdereus
2 posts since
Oct 20, 2009

Currently Being Moderated

14. Nov 2, 2009 3:37 PM (in response to Ryan Matte)
Re: zenstatus issues after 2.4.5 -> 2.5 upgrade

Yes, I did and still do.

Report Abuse

Like (0)

1 2 Previous Next

Go to original post

Legend

Correct Answers - 4 points
Helpful Answers - 2 points

Oct 28, 2009 6:37 PM

zenstatus issues after 2.4.5 -> 2.5 upgrade

Actions

More Like This

Incoming Links

Legend