Issues with 2.5.0

Up to Discussions in zenoss-users

102447 Views 10 Replies Latest reply: Feb 19, 2010 12:42 PM by cgriebel

cgriebel

175 posts since
Nov 29, 2007

Currently Being Moderated

Oct 29, 2009 7:10 AM

Long time user just upgraded yesterday from 2.4.5 to 2.5.0 and having major problems:

zenactions spewing errors:

2009-10-29 07:02:45,989 ERROR zen.ZenActions: unexpected exception
Traceback (most recent call last):
File "/opt/zenoss/Products/ZenEvents/zenactions.py", line 564, in runCycle
    self.mainbody()
File "/opt/zenoss/Products/ZenEvents/zenactions.py", line 557, in mainbody
    self.heartbeatEvents()
File "/opt/zenoss/Products/ZenEvents/zenactions.py", line 488, in heartbeatEvents
    hostname = self.fetchMonitorHostname(monitor)
File "/opt/zenoss/Products/ZenEvents/zenactions.py", line 463, in fetchMonitorHostname
    hostname = self.dmd.Monitors.getPerformanceMonitor(monitor).hostname
AttributeError: hostname

zenstatus and zencommand won't start

here's tail of zenstatus.log

2009-10-29 06:50:31,509 INFO zen.ZenStatus: Connecting to localhost:8789
2009-10-29 06:50:31,612 INFO zen.ZenStatus: Connected to ZenHub
2009-10-29 06:50:31,622 INFO zen.ZenStatus: Fetching property items
2009-10-29 06:50:44,683 INFO zen.ZenStatus: Fetching default RRDCreateCommand
2009-10-29 06:50:44,732 INFO zen.ZenStatus: Getting threshold classes
2009-10-29 06:50:44,781 INFO zen.ZenStatus: Getting collector thresholds
2009-10-29 06:50:51,064 ERROR zen.ZenStatus: Remote exception: Products.ZenHub.PBDaemon.RemoteException: Remote exception: zExceptions.NotFound: tcp_00443Traceback (most recent call last):
File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 75, in inner
    return callable(*args, **kw)
File "/opt/zenoss/Products/ZenHub/services/StatusConfig.py", line 66, in remote_services
    for svc in dev.getMonitoredComponents(collector='zenstatus'):
File "/opt/zenoss/Products/ZenModel/Device.py", line 593, in getMonitoredComponents
    collector=collector, type=type)
File "/opt/zenoss/Products/ZenModel/Device.py", line 623, in getDeviceComponents
    return [c.getObject() for c in brains]
File "/opt/zenoss/lib/python/Products/ZCatalog/CatalogBrains.py", line 92, in getObject
    target = parent.restrictedTraverse(path[-1])
File "/opt/zenoss/lib/python/OFS/Traversable.py", line 301, in restrictedTraverse
    return self.unrestrictedTraverse(path, default, restricted=True)
File "/opt/zenoss/lib/python/OFS/Traversable.py", line 284, in unrestrictedTraverse
    raise e
NotFound: tcp_00443
Traceback (most recent call last):
File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 75, in inner
    return callable(*args, **kw)
File "/opt/zenoss/Products/ZenHub/zenhubworker.py", line 116, in remote_execute
    return runOnce()
File "/opt/zenoss/Products/ZenHub/zenhubworker.py", line 111, in runOnce
    res = m(*args, **kw)
File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 83, in inner
    traceback.format_exc())
RemoteException: Remote exception: zExceptions.NotFound: tcp_00443Traceback (most recent call last):
File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 75, in inner
    return callable(*args, **kw)
File "/opt/zenoss/Products/ZenHub/services/StatusConfig.py", line 66, in remote_services
    for svc in dev.getMonitoredComponents(collector='zenstatus'):
File "/opt/zenoss/Products/ZenModel/Device.py", line 593, in getMonitoredComponents
    collector=collector, type=type)
File "/opt/zenoss/Products/ZenModel/Device.py", line 623, in getDeviceComponents
    return [c.getObject() for c in brains]
File "/opt/zenoss/lib/python/Products/ZCatalog/CatalogBrains.py", line 92, in getObject
    target = parent.restrictedTraverse(path[-1])
File "/opt/zenoss/lib/python/OFS/Traversable.py", line 301, in restrictedTraverse
    return self.unrestrictedTraverse(path, default, restricted=True)
File "/opt/zenoss/lib/python/OFS/Traversable.py", line 284, in unrestrictedTraverse
    raise e
NotFound: tcp_00443

2009-10-29 06:50:51,498 INFO zen.ZenStatus: Deleting PID file /opt/zenoss/var/zenstatus-localhost.pid ...
2009-10-29 06:50:51,499 INFO zen.ZenStatus: Daemon ZenStatus shutting down

So I thought maybe a zendmd and reindex() would be in order but I can't even run zendmd (running as user zenoss of course):

$ zendmd
2009-10-29 06:59:07 WARNING Application Duplicate Product name: After loading Product 'AdvancedQuery' from '/opt/zenoss/Products', I skipped the one in '/opt/zenoss/lib/python/Products'.
2009-10-29 06:59:07 WARNING Application Duplicate Product name: After loading Product 'CMFCore' from '/opt/zenoss/Products', I skipped the one in '/opt/zenoss/lib/python/Products'.
2009-10-29 06:59:07 WARNING Application Duplicate Product name: After loading Product 'Five' from '/opt/zenoss/Products', I skipped the one in '/opt/zenoss/lib/python/Products'.
2009-10-29 06:59:07 WARNING Application Duplicate Product name: After loading Product 'GenericSetup' from '/opt/zenoss/Products', I skipped the one in '/opt/zenoss/lib/python/Products'.
2009-10-29 06:59:07 WARNING Application Duplicate Product name: After loading Product 'ManagableIndex' from '/opt/zenoss/Products', I skipped the one in '/opt/zenoss/lib/python/Products'.
2009-10-29 06:59:07 WARNING Application Duplicate Product name: After loading Product 'OFolder' from '/opt/zenoss/Products', I skipped the one in '/opt/zenoss/lib/python/Products'.
2009-10-29 06:59:07 WARNING Application Duplicate Product name: After loading Product 'PluggableAuthService' from '/opt/zenoss/Products', I skipped the one in '/opt/zenoss/lib/python/Products'.
2009-10-29 06:59:07 WARNING Application Duplicate Product name: After loading Product 'PluginRegistry' from '/opt/zenoss/Products', I skipped the one in '/opt/zenoss/lib/python/Products'.
2009-10-29 06:59:08 ERROR ZODB.lock_file Error locking file /opt/zenoss/var/zeo1-1.zec.lock; pid=UNKNOWN
Traceback (most recent call last):
File "/opt/zenoss/lib/python/ZODB/lock_file.py", line 80, in __init__
    _lock_file(fp)
File "/opt/zenoss/lib/python/ZODB/lock_file.py", line 59, in _lock_file
    raise LockError("Couldn't lock %r" % file.name)
LockError: Couldn't lock '/opt/zenoss/var/zeo1-1.zec.lock'
Traceback (most recent call last):
File "/opt/zenoss/Products/ZenModel/zendmd.py", line 245, in ?
    myconsole = HistoryConsole(locals=_customStuff())
File "/opt/zenoss/Products/ZenModel/zendmd.py", line 71, in _customStuff
    app = Zope2.app()
File "/opt/zenoss/lib/python/Zope2/__init__.py", line 51, in app
    startup()
File "/opt/zenoss/lib/python/Zope2/__init__.py", line 47, in startup
    _startup()
File "/opt/zenoss/lib/python/Zope2/App/startup.py", line 65, in startup
    DB = dbtab.getDatabase('/', is_root=1)
File "/opt/zenoss/lib/python/Zope2/Startup/datatypes.py", line 288, in getDatabase
    db = factory.open(name, self.databases)
File "/opt/zenoss/lib/python/Zope2/Startup/datatypes.py", line 186, in open
    DB = self.createDB(database_name, databases)
File "/opt/zenoss/lib/python/Zope2/Startup/datatypes.py", line 183, in createDB
    return ZODBDatabase.open(self, databases)
File "/opt/zenoss/lib/python/ZODB/config.py", line 97, in open
    storage = section.storage.open()
File "/opt/zenoss/lib/python/ZODB/config.py", line 168, in open
    realm=self.config.realm)
File "/opt/zenoss/lib/python/ZEO/ClientStorage.py", line 340, in __init__
    self._cache = self.ClientCacheClass(cache_path, size=cache_size)
File "/opt/zenoss/lib/python/ZEO/cache.py", line 186, in __init__
    self._lock_file = ZODB.lock_file.LockFile(path + '.lock')
File "/opt/zenoss/lib/python/ZODB/lock_file.py", line 80, in __init__
    _lock_file(fp)
File "/opt/zenoss/lib/python/ZODB/lock_file.py", line 59, in _lock_file
    raise LockError("Couldn't lock %r" % file.name)
ZODB.lock_file.LockError: Couldn't lock '/opt/zenoss/var/zeo1-1.zec.lock'

I have stopped all zenoss processes, deleted /opt/zenoss/var/*zec* -- no luck.

Any help much appreciated.

Chock

Like (0)

Captain MESO
114 posts since
Mar 5, 2008

Currently Being Moderated

1. Oct 29, 2009 11:32 AM (in response to cgriebel)
Re: Issues with 2.5.0

Seems to be that you have the same problem after upgrading. Check thread/11846?tstart=0

Report Abuse

Like (0)
cgriebel
175 posts since
Nov 29, 2007

Currently Being Moderated

2. Oct 29, 2009 1:20 PM (in response to Captain MESO)
Re: Issues with 2.5.0

Ok. I'll focus on the issue of zendmd not running (I'd like to be able to do a Zope reindex()). I edited/opt/zenoss/lib/python/ZODB/lock_file.py and commented out the try/except so I could see the error that flock was raising:
    def _lock_file(file):
        #try:
        fcntl.flock(file.fileno(), _flags)
        #except IOError:
        #    raise LockError("Couldn't lock %r" % file.name)

restart zenoss, ran zendmd
and got this:
[some stuff deleted]
Traceback (most recent call last):
File "/opt/zenoss/lib/python/ZODB/lock_file.py", line 80, in __init__
    _lock_file(fp)
File "/opt/zenoss/lib/python/ZODB/lock_file.py", line 57, in _lock_file
    fcntl.flock(file.fileno(), _flags)
IOError: [Errno 11] Resource temporarily unavailable
Traceback (most recent call last):
File "/opt/zenoss/Products/ZenModel/zendmd.py", line 245, in ?
    myconsole = HistoryConsole(locals=_customStuff())
File "/opt/zenoss/Products/ZenModel/zendmd.py", line 71, in _customStuff
    app = Zope2.app()
File "/opt/zenoss/lib/python/Zope2/__init__.py", line 51, in app
    startup()
File "/opt/zenoss/lib/python/Zope2/__init__.py", line 47, in startup
    _startup()
File "/opt/zenoss/lib/python/Zope2/App/startup.py", line 65, in startup
    DB = dbtab.getDatabase('/', is_root=1)
File "/opt/zenoss/lib/python/Zope2/Startup/datatypes.py", line 288, in getDatabase
    db = factory.open(name, self.databases)
File "/opt/zenoss/lib/python/Zope2/Startup/datatypes.py", line 186, in open
    DB = self.createDB(database_name, databases)
File "/opt/zenoss/lib/python/Zope2/Startup/datatypes.py", line 183, in createDB
    return ZODBDatabase.open(self, databases)
File "/opt/zenoss/lib/python/ZODB/config.py", line 97, in open
    storage = section.storage.open()
File "/opt/zenoss/lib/python/ZODB/config.py", line 168, in open
    realm=self.config.realm)
File "/opt/zenoss/lib/python/ZEO/ClientStorage.py", line 340, in __init__
    self._cache = self.ClientCacheClass(cache_path, size=cache_size)
File "/opt/zenoss/lib/python/ZEO/cache.py", line 186, in __init__
    self._lock_file = ZODB.lock_file.LockFile(path + '.lock')
File "/opt/zenoss/lib/python/ZODB/lock_file.py", line 80, in __init__
    _lock_file(fp)
File "/opt/zenoss/lib/python/ZODB/lock_file.py", line 57, in _lock_file
    fcntl.flock(file.fileno(), _flags)
IOError: [Errno 11] Resource temporarily unavailable

So, another process has hold of the lock file.

$ lsof /opt/zenoss/var/zeo1-1.zec.lock
COMMAND   PID   USER   FD   TYPE DEVICE SIZE   NODE NAME
python 22439 zenoss    8uW REG 253,0    0 756266 /opt/zenoss/var/zeo1-1.zec.lock

[Edit: For clarification, below is the process information for the above pid (22439) returned by lsof]

0 S zenoss   22439 22437 2 76   0 - 26769 -      09:52 ?        00:00:05 /opt/zenoss/bin/python /opt/zenoss/lib/python/Zope2/Startup/run.py -C /opt/zenoss/etc/zope.conf

It's zope's run.py. Should that run.py be holding on to this lock? I'm confused and not sure what's changed. Can others run zendmd OK in 2.5.0?

Chock

Report Abuse

Like (0)
cgriebel
175 posts since
Nov 29, 2007

Currently Being Moderated

3. Nov 2, 2009 1:02 PM (in response to cgriebel)
Re: 2.5.0 - zendmd won't run

Anyone have any ideas on what's causing the EAGAIN lock error ("Resource temporarily unavailable") error when I try to run zendmd? (described in previous post)

Report Abuse

Like (0)
Ryan Matte
653 posts since
Mar 26, 2009

Currently Being Moderated

4. Nov 2, 2009 6:43 PM (in response to cgriebel)
Re: 2.5.0 - zendmd won't run

If you had the WMI Data Source ZenPack installed when you upgraded then that's your issue. If not, then I'm not sure what's causing that.

Report Abuse

Like (0)
cgriebel
175 posts since
Nov 29, 2007

Currently Being Moderated

5. Nov 3, 2009 6:31 AM (in response to Ryan Matte)
Re: 2.5.0 - zendmd won't run

No, I don't have that zenpack installed. I haven't heard of anyone else having the problem with zendmd not running because it can't lock the file that zope has a lock on. I've never had this problem before. I wonder how many people have even tried to run zendmd with 2.5.0 yet.

Chock

Report Abuse

Like (0)
Ryan Matte
653 posts since
Mar 26, 2009

Currently Being Moderated

6. Nov 3, 2009 9:06 AM (in response to cgriebel)
Re: 2.5.0 - zendmd won't run

I've used zendmd with 2.5 on numerous occasions and have not come across any issues. You might want to check your file permissions in your $ZENHOME directory.

Report Abuse

Like (0)
cgriebel
175 posts since
Nov 29, 2007

Currently Being Moderated

7. Nov 3, 2009 10:21 AM (in response to Ryan Matte)
Re: 2.5.0 - zendmd won't run

I'm glad to hear that zendmd works in 2.5.0

I think my perm's are good:
[zenoss:/opt/zenoss]$ pwd
/opt/zenoss
[zenoss:/opt/zenoss]$ ls -lad . var var/zeo1-1.zec.lock
drwxr-xr-x 26 zenoss zenoss 4096 Nov 2 10:36 .
drwxrwxr-x   7 zenoss zenoss 4096 Nov 2 10:35 var
-rw-rw-r--   1 zenoss zenoss    0 Nov 2 12:44 var/zeo1-1.zec.lock

Would you mind checking on your system like I've done below to see if Zope has a lock on this file like mine does:
[root:~]# lsof /opt/zenoss/var/zeo1-1.zec.lock
COMMAND   PID   USER   FD   TYPE DEVICE SIZE   NODE NAME
python 22439 zenoss    8uW REG 253,0    0 756266 /opt/zenoss/var/zeo1-1.zec.lock
[root~]# ps wp 22439
PID TTY      STAT   TIME COMMAND
22439 ?        Sl    30:21 /opt/zenoss/bin/python /opt/zenoss/lib/python/Zope2/Startup/run.py -C /opt/zenoss/etc/zope.conf

I appreciate your help and input.

Chock

Report Abuse

Like (0)
cgriebel
175 posts since
Nov 29, 2007

Currently Being Moderated

8. Nov 6, 2009 9:21 AM (in response to cgriebel)
Re: 2.5.0 - zendmd won't run

I found a workaround on the issue of zendmd not running b/c of zope having file locked (here's the pertinent error message again for those interested):
2009-11-06 09:14:52 ERROR ZODB.lock_file Error locking file /opt/zenoss/var/zeo1-1.zec.lock; pid=UNKNOWN

I was able to run zendmd if I stopped Zope first. Got my reindex and commit done and now all my daemons are starting. Seems healthy for the moment.

I can live with this workaround.

Chock

Report Abuse

Like (0)
Andrea Consadori
863 posts since
Feb 11, 2008

Currently Being Moderated

9. Nov 8, 2009 2:39 AM (in response to cgriebel)
Re: 2.5.0 - zendmd won't run

ok but if i restart zenoss the problem persist...

i open a ticket

http://dev.zenoss.org/trac/ticket/5798

Report Abuse

Like (0)
cgriebel
175 posts since
Nov 29, 2007

Currently Being Moderated

10. Feb 19, 2010 12:42 PM (in response to Andrea Consadori)
Re: 2.5.0 - zendmd won't run

I see a simple fix was recently posted to your ticket (http://dev.zenoss.org/trac/ticket/5798)
I changed the zope.conf per the instructions and now I can run zendmd just fine while zope is running.

All better!!

Chock

Report Abuse

Like (0)

Go to original post

Legend

Correct Answers - 4 points
Helpful Answers - 2 points

Oct 29, 2009 7:10 AM

Issues with 2.5.0

Actions

More Like This

Legend