Archived community.zenoss.org | full text search
Skip navigation
Up to Discussions in zenoss-dev
44357 Views 5 Replies Latest reply: Jan 21, 2010 2:13 PM by chickenandbeans RSS
chickenandbeans Rank: White Belt
chickenandbeans
11 posts since
Aug 11, 2009
Currently Being Moderated

Jan 12, 2010 9:32 AM

dmd = ZenScriptBase(connect=True).dmd multiple connections

I've written a few scripts using dmd = ZenScriptBase(connect=True).dmd, let's call them script1.py and script2.py.

I'm trying to pipe the output of script1 to script2. This results in a conflict error:

 

 

 

Traceback (most recent call last):
  File "./script1.py", line 13, in ?
    dmd = ZenScriptBase(connect=True,noopts=True).dmd
  File "/opt/zenoss/zenoss/Products/ZenUtils/ZenScriptBase.py", line 39, in __init__
    self.connect()
  File "/opt/zenoss/zenoss/Products/ZenUtils/ZenScriptBase.py", line 57, in connect
    commit()
  File "/opt/zenoss/zenoss/lib/python/transaction/_manager.py", line 93, in commit
    return self.get().commit()
  File "/opt/zenoss/zenoss/lib/python/transaction/_transaction.py", line 325, in commit
    self._commitResources()
  File "/opt/zenoss/zenoss/lib/python/transaction/_transaction.py", line 424, in _commitResources
    rm.commit(self)
  File "/opt/zenoss/zenoss/lib/python/ZODB/Connection.py", line 541, in commit
    self._commit(transaction)
  File "/opt/zenoss/zenoss/lib/python/ZODB/Connection.py", line 578, in _commit
    raise ConflictError(object=obj)
ZODB.POSException.ConflictError: database conflict error (oid 0x0cd5, class Products.ZenModel.DataRoot.DataRoot)

 

 

*nix box:

COMMAND 1: script1.py | script2.py = db conflict error

COMMAND2: script1.py > script1.out && cat script1.out | script2.py = works!

 

The difference between command 1 and command 2 is the number of connections, two and one respectively. Since the two scripts are connecting the same way... perhaps it's connection/locking issue? Has anyone created a zendmd script that accepts piped data in this way?

Tags: zendmd, zenscriptbase
  • chickenandbeans Rank: White Belt
    chickenandbeans
    11 posts since
    Aug 11, 2009
    Currently Being Moderated
    1. Jan 19, 2010 11:04 AM (in response to chickenandbeans)
    Re: dmd = ZenScriptBase(connect=True).dmd multiple connections

    Am I to believe that one cannot have multiple connections using the dmd = ZenScriptBase(connect=True,noopts=True).dmd ? Is there another way to connect that allows more than one?

    To check for locks?

     

     

    edit:

     

    Another thing, both scripts are just printing out values... no writes :/ ...weird huh?

  • chickenandbeans Rank: White Belt
    chickenandbeans
    11 posts since
    Aug 11, 2009
    Currently Being Moderated
    2. Jan 19, 2010 11:38 AM (in response to chickenandbeans)
    Re: dmd = ZenScriptBase(connect=True).dmd multiple connections

    Status update.

     

     

    Zenoss Version: 2.5.1

    Zope Version: 2.11.2

     

    Been talking on IRC with 'rocket'. and he suggested commenting out the "commit" on line 57 in ZenScript /opt/zenoss/zenoss/Products/ZenUtils. This works!

    But this cannot be used in anger yet...

     

    1. Why does this work?

    The issue may be related to the following ticket:

    http://dev.zenoss.com/trac/ticket/5128

     

    commits being called at the same time causing issues? But would that even be possible, mutex problems?

     

     

    2. What effect doing this would have on other functions in Zenoss?

    I notice the commit happens after a test:

     

    if not getattr(self.dmd, 'propertyTransformers', None):
                self.dmd.propertyTransformers = {}
                commit()

     

    What happens if this property is not commited at that point?

     

    Thoughts?

  • mrchippy ZenossEmployee
    mrchippy
    34 posts since
    May 12, 2008
    Currently Being Moderated
    3. Jan 21, 2010 12:25 PM (in response to chickenandbeans)
    Re: dmd = ZenScriptBase(connect=True).dmd multiple connections

    Conflict errors arise because two things are done to the database and the database doesn't know how to determine the "winner".  For example, if you have two connections open and one sets a = 1, then the other sets a = 2 and issues a commit().  If the first connection tries to commit now, the conflict exception gets thrown because the db doesn't know how to make sure data will be consistent.  There are three ways to try to avoid this in this simple example-- (1) issue a commit on connection one before connection two tries to modify the object; (2) issue a sync() call before changing the object in connection 2 (sync replaces your local objects with their current state in the db); (3) don't commit anything.  It sounds like #3 worked for you in your script.  Did you not need anything in the database to change?  Was it only retrieving information from the db?

     

    In your original message, the scripta | scriptb fails because the connections are open simultaneously.   In the second, the first script closes its connection before the second one starts.

     

    Can you post the scripts?

  • chickenandbeans Rank: White Belt
    chickenandbeans
    11 posts since
    Aug 11, 2009
    Currently Being Moderated
    4. Jan 21, 2010 1:23 PM (in response to mrchippy)
    Re: dmd = ZenScriptBase(connect=True).dmd multiple connections

    Right after an extreamly helpful and informative chat on irc with mrchippy...

     

     

     

    While editing the code above worked... it's probably a bad idea to make your own zenoss code base changes ! But it does show that the problem is with an errant commit. What we can do is override any commit using the following:

     

    import transaction
    def fakeCommit(*args, **kwargs):
        pass
    transaction.commit = fakeCommit

     

     

    Since my *simple* script is read only ( no commits() ) it is probably ok. If your script is commiting stuff.. THIS IS A BAD IDEA

  • chickenandbeans Rank: White Belt
    chickenandbeans
    11 posts since
    Aug 11, 2009
    Currently Being Moderated
    5. Jan 21, 2010 2:13 PM (in response to chickenandbeans)
    Re: dmd = ZenScriptBase(connect=True).dmd multiple connections

    Here's a snippet that caused the error. Imports in both scripts are identical:

     

     

    import Globals
    from Products.ZenUtils.ZenScriptBase import ZenScriptBase
    import sys
    from optparse import OptionParser
    dmd = ZenScriptBase(connect=True,noopts=True).dmd

     

     

    for template in fullPath:
            templateName=template.viewName()

     

    print "dmd.Devices.manage_addRRDTemplate(\""+templateName+"\")"
            templateName="_getOb(\'"+template.viewName()+"\')"
            print "dmd.Devices.rrdTemplates."+templateName+".description=\""+template.getSubObjects()[0].description+"\""
            print "dmd.Devices.rrdTemplates."+templateName+".targetPythonClass=\""+template.getSubObjects()[0].targetPythonClass+"\""
        #   #Template datasources
            #for templateDataSource in template.datasources():
            for templateDataSource in template.getRRDDataSources():
                # "BasicDataSource" is hardcoded here because currently it's the only one used
                #  Unlikely that another type would be used
                templateDataSourceName=templateDataSource.viewName()
                print "dmd.Devices.rrdTemplates."+templateName+".manage_addRRDDataSource(\""+templateDataSourceName+"\",\"BasicDataSource."+templateDataSource.so
    urcetype+"\")"
                templateDataSourceName="_getOb(\'"+templateDataSource.viewName()+"\')"
                #Recurse into template datasources
                print "dmd.Devices.rrdTemplates."+templateName+".datasources."+templateDataSourceName+".sourcetype=\""+templateDataSource.sourcetype+"\""

     

                if templateDataSource.enabled:
                    sourceEnabled='True'
                else:
                    sourceEnabled = 'False'

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points