Jan 12, 2010 9:32 AM
dmd = ZenScriptBase(connect=True).dmd multiple connections
-
Like (0)
I've written a few scripts using dmd = ZenScriptBase(connect=True).dmd, let's call them script1.py and script2.py.
I'm trying to pipe the output of script1 to script2. This results in a conflict error:
Traceback (most recent call last):
File "./script1.py", line 13, in ?
dmd = ZenScriptBase(connect=True,noopts=True).dmd
File "/opt/zenoss/zenoss/Products/ZenUtils/ZenScriptBase.py", line 39, in __init__
self.connect()
File "/opt/zenoss/zenoss/Products/ZenUtils/ZenScriptBase.py", line 57, in connect
commit()
File "/opt/zenoss/zenoss/lib/python/transaction/_manager.py", line 93, in commit
return self.get().commit()
File "/opt/zenoss/zenoss/lib/python/transaction/_transaction.py", line 325, in commit
self._commitResources()
File "/opt/zenoss/zenoss/lib/python/transaction/_transaction.py", line 424, in _commitResources
rm.commit(self)
File "/opt/zenoss/zenoss/lib/python/ZODB/Connection.py", line 541, in commit
self._commit(transaction)
File "/opt/zenoss/zenoss/lib/python/ZODB/Connection.py", line 578, in _commit
raise ConflictError(object=obj)
ZODB.POSException.ConflictError: database conflict error (oid 0x0cd5, class Products.ZenModel.DataRoot.DataRoot)
*nix box:
COMMAND 1: script1.py | script2.py = db conflict error
COMMAND2: script1.py > script1.out && cat script1.out | script2.py = works!
The difference between command 1 and command 2 is the number of connections, two and one respectively. Since the two scripts are connecting the same way... perhaps it's connection/locking issue? Has anyone created a zendmd script that accepts piped data in this way?
Am I to believe that one cannot have multiple connections using the dmd = ZenScriptBase(connect=True,noopts=True).dmd ? Is there another way to connect that allows more than one?
To check for locks?
edit:
Another thing, both scripts are just printing out values... no writes :/ ...weird huh?
Status update.
Zenoss Version: 2.5.1
Zope Version: 2.11.2
Been talking on IRC with 'rocket'. and he suggested commenting out the "commit" on line 57 in ZenScript /opt/zenoss/zenoss/Products/ZenUtils. This works!
But this cannot be used in anger yet...
1. Why does this work?
The issue may be related to the following ticket:
http://dev.zenoss.com/trac/ticket/5128
commits being called at the same time causing issues? But would that even be possible, mutex problems?
2. What effect doing this would have on other functions in Zenoss?
I notice the commit happens after a test:
if not getattr(self.dmd, 'propertyTransformers', None):
self.dmd.propertyTransformers = {}
commit()
What happens if this property is not commited at that point?
Thoughts?
Conflict errors arise because two things are done to the database and the database doesn't know how to determine the "winner". For example, if you have two connections open and one sets a = 1, then the other sets a = 2 and issues a commit(). If the first connection tries to commit now, the conflict exception gets thrown because the db doesn't know how to make sure data will be consistent. There are three ways to try to avoid this in this simple example-- (1) issue a commit on connection one before connection two tries to modify the object; (2) issue a sync() call before changing the object in connection 2 (sync replaces your local objects with their current state in the db); (3) don't commit anything. It sounds like #3 worked for you in your script. Did you not need anything in the database to change? Was it only retrieving information from the db?
In your original message, the scripta | scriptb fails because the connections are open simultaneously. In the second, the first script closes its connection before the second one starts.
Can you post the scripts?
Right after an extreamly helpful and informative chat on irc with mrchippy...
While editing the code above worked... it's probably a bad idea to make your own zenoss code base changes ! But it does show that the problem is with an errant commit. What we can do is override any commit using the following:
import transaction
def fakeCommit(*args, **kwargs):
pass
transaction.commit = fakeCommit
Since my *simple* script is read only ( no commits() ) it is probably ok. If your script is commiting stuff.. THIS IS A BAD IDEA
Here's a snippet that caused the error. Imports in both scripts are identical:
import Globals
from Products.ZenUtils.ZenScriptBase import ZenScriptBase
import sys
from optparse import OptionParser
dmd = ZenScriptBase(connect=True,noopts=True).dmd
for template in fullPath:
templateName=template.viewName()
print "dmd.Devices.manage_addRRDTemplate(\""+templateName+"\")"
templateName="_getOb(\'"+template.viewName()+"\')"
print "dmd.Devices.rrdTemplates."+templateName+".description=\""+template.getSubObjects()[0].description+"\""
print "dmd.Devices.rrdTemplates."+templateName+".targetPythonClass=\""+template.getSubObjects()[0].targetPythonClass+"\""
# #Template datasources
#for templateDataSource in template.datasources():
for templateDataSource in template.getRRDDataSources():
# "BasicDataSource" is hardcoded here because currently it's the only one used
# Unlikely that another type would be used
templateDataSourceName=templateDataSource.viewName()
print "dmd.Devices.rrdTemplates."+templateName+".manage_addRRDDataSource(\""+templateDataSourceName+"\",\"BasicDataSource."+templateDataSource.so
urcetype+"\")"
templateDataSourceName="_getOb(\'"+templateDataSource.viewName()+"\')"
#Recurse into template datasources
print "dmd.Devices.rrdTemplates."+templateName+".datasources."+templateDataSourceName+".sourcetype=\""+templateDataSource.sourcetype+"\""
if templateDataSource.enabled:
sourceEnabled='True'
else:
sourceEnabled = 'False'
Follow Us On Twitter »
|
Latest from the Zenoss Blog » | Community | Products | Services Resources | Customers Partners | About Us | ||
Copyright © 2005-2011 Zenoss, Inc.
|
||||||||