Oct 28, 2009 6:37 PM
zenstatus issues after 2.4.5 -> 2.5 upgrade
-
Like (0)
I upgraded my stack install of 2.4.5 to 2.5 today and after fixing a few other minor problems I still have one remaining. Both my primary and secondary collector show zenstatus as having issues, and the log is full of messages like the following:
2009-10-28 15:30:53,394 WARNING zen.ZenStatus: Scan cycle not complete in 119.99 seconds
2009-10-28 15:30:53,394 WARNING zen.ZenStatus: Waiting for the cycle to complete
2009-10-28 15:31:53,398 WARNING zen.ZenStatus: Scan cycle not complete in 179.99 seconds
2009-10-28 15:31:53,398 WARNING zen.ZenStatus: Restarting jobs for another cycle
2009-10-28 15:32:53,402 WARNING zen.ZenStatus: Scan cycle not complete in 59.98 seconds
2009-10-28 15:32:53,402 WARNING zen.ZenStatus: Waiting for the cycle to complete
2009-10-28 15:33:53,402 WARNING zen.ZenStatus: Scan cycle not complete in 119.98 seconds
2009-10-28 15:33:53,402 WARNING zen.ZenStatus: Waiting for the cycle to complete
2009-10-28 15:34:53,406 WARNING zen.ZenStatus: Scan cycle not complete in 179.99 seconds
2009-10-28 15:34:53,406 WARNING zen.ZenStatus: Restarting jobs for another cycle
2009-10-28 15:35:53,410 WARNING zen.ZenStatus: Scan cycle not complete in 59.99 seconds
2009-10-28 15:35:53,410 WARNING zen.ZenStatus: Waiting for the cycle to complete
I don't have a solution but can confirm this problem. I have exactly the same issue since upgrading from 2.4.5 to 2.5. Any hints?
2009-10-28 21:41:42,548 INFO zen.ZenStatus: Connected to ZenHub
2009-10-28 21:41:42,582 INFO zen.ZenStatus: Fetching property items
2009-10-28 21:42:42,311 INFO zen.ZenStatus: Fetching default RRDCreateCommand
2009-10-28 21:42:43,550 INFO zen.ZenStatus: Getting threshold classes
2009-10-28 21:42:45,713 INFO zen.ZenStatus: Getting collector thresholds
2009-10-28 21:45:31,860 WARNING zen.ZenStatus: Scan cycle not complete in 32.23 seconds
2009-10-28 21:45:31,868 WARNING zen.ZenStatus: Waiting for the cycle to complete
2009-10-28 21:47:31,901 WARNING zen.ZenStatus: Scan cycle not complete in 152.27 seconds
2009-10-28 21:47:31,904 WARNING zen.ZenStatus: Waiting for the cycle to complete
2009-10-28 21:49:31,965 WARNING zen.ZenStatus: Scan cycle not complete in 272.34 seconds
2009-10-28 21:49:31,967 WARNING zen.ZenStatus: Restarting jobs for another cycle
2009-10-28 21:51:31,977 WARNING zen.ZenStatus: Scan cycle not complete in 79.62 seconds
2009-10-28 21:51:31,980 WARNING zen.ZenStatus: Waiting for the cycle to complete
2009-10-28 21:53:31,982 WARNING zen.ZenStatus: Scan cycle not complete in 199.63 seconds
2009-10-28 21:53:31,982 WARNING zen.ZenStatus: Waiting for the cycle to complete
2009-10-28 21:55:32,081 WARNING zen.ZenStatus: Scan cycle not complete in 319.73 seconds
2009-10-28 21:55:32,086 WARNING zen.ZenStatus: Restarting jobs for another cycle
2009-10-28 21:57:32,085 WARNING zen.ZenStatus: Scan cycle not complete in 112.45 seconds
2009-10-28 21:57:32,086 WARNING zen.ZenStatus: Waiting for the cycle to complete
Thanks guyverix. Verbose logs for zenstatus and zenhub seems to be ok for me. But zenaction logging is very interesting. Since upgrading it shows:
2009-10-28 21:24:09,744 INFO zen.ZenActions: Processed 0 commands in 0.005499 2009-10-28 21:24:10,291 ERROR zen.ZenActions: unexpected exception Traceback (most recent call last): File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle self.mainbody() File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 557, in mainbody self.heartbeatEvents() File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 488, in heartbeatEvents hostname = self.fetchMonitorHostname(monitor) File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 463, in fetchMonitorHostname hostname = self.dmd.Monitors.getPerformanceMonitor(monitor).hostname
AttributeError: hostname
I think this is causing the zenstatus problem. What could it be?
Verbose log for zenaction:
2009-10-29 06:13:56,610 INFO zen.ZenActions: Processed 0 commands in 0.000154
2009-10-29 06:13:56,611 DEBUG zen.ZenActions: call age_events(96, 5);
2009-10-29 06:13:56,612 DEBUG zen.DbConnectionPool: Retrieved a connection; Pool size: 0
2009-10-29 06:13:56,617 DEBUG zen.DbConnectionPool: Returned a connection; Pool size: 1
2009-10-29 06:13:56,618 DEBUG zen.ZenActions: SELECT monitor, component FROM status WHERE eventClass = '/Status/Heartbeat'
2009-10-29 06:13:56,618 DEBUG zen.DbConnectionPool: Retrieved a connection; Pool size: 0
2009-10-29 06:13:56,630 DEBUG zen.DbConnectionPool: Returned a connection; Pool size: 1
2009-10-29 06:13:56,630 DEBUG zen.ZenActions: SELECT device, component FROM heartbeat WHERE DATE_ADD(lastTime, INTERVAL timeout SECOND) <= NOW();
2009-10-29 06:13:56,630 DEBUG zen.DbConnectionPool: Retrieved a connection; Pool size: 0
2009-10-29 06:13:56,638 DEBUG zen.DbConnectionPool: Returned a connection; Pool size: 1
2009-10-29 06:13:56,639 ERROR zen.ZenActions: unexpected exception
Traceback (most recent call last):
File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle
self.mainbody()
File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 557, in mainbody
self.heartbeatEvents()
File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 488, in heartbeatEvents
hostname = self.fetchMonitorHostname(monitor)
File "/usr/local/zenoss/zenoss/Products/ZenEvents/zenactions.py", line 463, in fetchMonitorHostname
hostname = self.dmd.Monitors.getPerformanceMonitor(monitor).hostname
AttributeError: hostname
I too am getting the same errors in zenstatus and zenactions which have occurred from the moment of upgrading from 2.4.5 to 2.5.
Thanks a lot kisielk. 5714 fixed zenactions.
Zenstatus problem still remains after 2.4.5 --> 2.5 updgrade. Any solution available for this problem?
I am also seeing the issue that zenstatus.log is crashing every minute. This occurs on the 2.5 VMware image...
2009-10-30 07:34:33,535 ERROR zen.ZenActions: unexpected exception
Traceback (most recent call last):
File "/home/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle
self.mainbody()
File "/home/zenoss/Products/ZenEvents/zenactions.py", line 557, in mainbody
self.heartbeatEvents()
File "/home/zenoss/Products/ZenEvents/zenactions.py", line 488, in heartbeatEvents
hostname = self.fetchMonitorHostname(monitor)
File "/home/zenoss/Products/ZenEvents/zenactions.py", line 463, in fetchMonitorHostname
hostname = self.dmd.Monitors.getPerformanceMonitor(monitor).hostname
AttributeError: hostname
as well as on our upgraded SUSE 10.2 production machine::
2009-10-30 08:34:01,683 ERROR zen.ZenActions: unexpected exception
Traceback (most recent call last):
File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle
self.mainbody()
File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 552, in mainbody
self.eventCommands(zem)
File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 541, in eventCommands
self.processEvent(zem, command, self.runEventCommand)
File "/usr/local/zenoss/Products/ZenEvents/zenactions.py", line 258, in processEvent
fields = context.getEventFields()
File "/usr/local/zenoss/Products/ZenEvents/EventCommand.py", line 63, in getEventFields
return self.eventManager.getFieldList()
AttributeError: eventManager
Replacing the file zenactions.py from changeset 15582 didn't help, the zenactions daemon won't even start with it (no error log entry)
Any ideas?
Martin
I've seen this happen before with zenmodeler. The issue ended up being that a value that shouldn't be stored to the object database did get stored. The fix in that case was to run "zendmd" as the zenoss user and paste in the following snippet.
del(dmd.Monitors.Performance.localhost._properties)
commit()
Chet,
that command doesn't seem to work on either of the two machines:
>>> del(dmd.Monitors.Performance.localhost._properties)
Traceback (most recent call last):
File "<console>", line 1, in ?
AttributeError: _properties
However, on my upgraded machine I found the following left over entry from helloWorld Zenpack:
/zport/dmd/ZenEventManager/commands/helloWorld
Could this be the culprit on my upgraded machine? If it is, how can I get rid of this dummy helloWorld object? (d_delObject('helloWorld') fails with "AttributeError: eventManager" and installation and removal of the old Zenpack doesn' t delete it either)
Martin
Thanks Chet, but I get the same error message:
>>> del(dmd.Monitors.Performance.localhost._properties)
Traceback (most recent call last):
File "<console>", line 1, in ?
AttributeError: _properties
Typo?
Follow Us On Twitter »
|
Latest from the Zenoss Blog » | Community | Products | Services Resources | Customers Partners | About Us | ||
Copyright © 2005-2011 Zenoss, Inc.
|
||||||||