Archived community.zenoss.org | full text search
Skip navigation
Up to Discussions in zenoss-users
4447 Views 13 Replies Latest reply: Sep 20, 2011 9:33 AM by gadams RSS
gadams Rank: White Belt
gadams
Currently Being Moderated

Sep 6, 2011 9:09 AM

Upgraded to 3.2, now broken

When upgrading from 3.1 to 3.2 on CentOS 5.6 I got the following error in install.log:

 

....

INFO:zen.migrate:Reindexing components.  This may take a while ...
INFO:zen.migrate:indexed 1 components
WARNING:zen.migrate:Recovering
Traceback (most recent call last):
  File "/opt/zenoss/Products/ZenModel/migrate/zenmigrate.py", line 21, in <module>
    main()
  File "/opt/zenoss/Products/ZenModel/migrate/zenmigrate.py", line 18, in main
    m.main()
  File "/opt/zenoss/Products/ZenModel/migrate/Migrate.py", line 417, in main
    self.cutover()
  File "/opt/zenoss/Products/ZenModel/migrate/Migrate.py", line 304, in cutover
    self.migrate()
  File "/opt/zenoss/Products/ZenModel/migrate/Migrate.py", line 280, in migrate
    m.cutover(self.dmd)
  File "/opt/zenoss/Products/ZenModel/migrate/eventfacadeUsingCatalog.py", line 53, in cutover
    notify(IndexingEvent(comp))
  File "/opt/zenoss/lib/python/zope/event/__init__.py", line 23, in notify
    subscriber(event)
  File "/opt/zenoss/lib/python/zope/component/event.py", line 26, in dispatch
    for ignored in zope.component.subscribers(event, None):
  File "/opt/zenoss/lib/python/zope/component/_api.py", line 138, in subscribers
    return sitemanager.subscribers(objects, interface)
  File "/opt/zenoss/lib/python/zope/component/registry.py", line 323, in subscribers
    return self.adapters.subscribers(objects, provided)
  File "/opt/zenoss/lib/python/zope/interface/adapter.py", line 579, in subscribers
    subscription(*objects)
  File "/opt/zenoss/lib/python/zope/component/event.py", line 33, in objectEventNotify
    adapters = zope.component.subscribers((event.object, event), None)
  File "/opt/zenoss/lib/python/zope/component/_api.py", line 138, in subscribers
    return sitemanager.subscribers(objects, interface)
  File "/opt/zenoss/lib/python/zope/component/registry.py", line 323, in subscribers
    return self.adapters.subscribers(objects, provided)
  File "/opt/zenoss/lib/python/zope/interface/adapter.py", line 579, in subscribers
    subscription(*objects)
  File "/opt/zenoss/Products/Zuul/catalog/events.py", line 52, in onIndexingEvent
    update_metadata=event.update_metadata)
  File "/opt/zenoss/Products/Zuul/catalog/global_catalog.py", line 322, in catalog_object
    ZCatalog.catalog_object(self, ob, uid, **kwargs)
  File "/opt/zenoss/lib/python/Products/ZCatalog/ZCatalog.py", line 529, in catalog_object
    update_metadata=update_metadata)
  File "/opt/zenoss/lib/python/Products/ZCatalog/Catalog.py", line 339, in catalogObject
    index = self.updateMetadata(object, uid)
  File "/opt/zenoss/lib/python/Products/ZCatalog/Catalog.py", line 278, in updateMetadata
    newDataRecord = self.recordify(object)
  File "/opt/zenoss/lib/python/Products/ZCatalog/Catalog.py", line 417, in recordify
    if(attr is not MV and safe_callable(attr)): attr=attr()
  File "/opt/zenoss/Products/Zuul/catalog/global_catalog.py", line 224, in searchExcerpt
    o.name(), o.getParentDeviceName())
TypeError: 'str' object is not callable
Stopping Zope Object Database
.
daemon process stopped
Zenoss upgrade pre-restart commands complete.
Daemon: zeoctl .
   ......



 

 

Now Zenoss 3.2 has started but neither Events console nor Infrastructure can be loaded in the browser, the "Unable to connect to the server." message is displayed.

Messages like this appearing in events.log:

 

2011-09-06T13:22:35 ERROR Zope.SiteErrorLog 1315311755.370.698218486027 http://ugbbsk1003.wsatkins.com:8080/zport/dmd/Events/evconsole_router
Traceback (innermost last):
  Module ZPublisher.Publish, line 127, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 47, in call_object
  Module Products.ZenUtils.extdirect.zope.router, line 24, in __call__
  Module Products.ZenUtils.extdirect.router, line 88, in __call__
  Module Products.ZenUtils.jsonutils, line 152, in json
  Module json, line 230, in dumps
  Module json.encoder, line 367, in encode
  Module json.encoder, line 309, in _iterencode
  Module json.encoder, line 275, in _iterencode_dict
  Module json.encoder, line 309, in _iterencode
  Module json.encoder, line 275, in _iterencode_dict
  Module json.encoder, line 306, in _iterencode
  Module json.encoder, line 204, in _iterencode_list
  Module json.encoder, line 309, in _iterencode
  Module json.encoder, line 275, in _iterencode_dict
  Module json.encoder, line 309, in _iterencode
  Module json.encoder, line 275, in _iterencode_dict
  Module json.encoder, line 317, in _iterencode
  Module json.encoder, line 323, in _iterencode_default
  Module json.encoder, line 344, in default
TypeError: <bound method DeviceClass.titleOrId of <DeviceClass at /zport/dmd/Devices>> is not JSON serializable

 

Message was edited by: gadams Downgraded rpms and used zenrestore, now seem to be running OK back at version 3.1 .

  • rphil001 Rank: White Belt
    rphil001
    7 posts since
    Jul 25, 2011
    Currently Being Moderated
    1. Sep 6, 2011 11:52 AM (in response to gadams)
    Re: Upgraded to 3.2, now broken

    Hi, we have exactly the same issue here. We think it may be a zenpack thats causing the issue, which ones did you have preinstall? I have list ours below in case of somen common ones

     

    ZenPacks.HP.Proliant.MIBs

    ZenPacks.SCC.ShowGraphPortlet

    ZenPacks.community.DistributedCollectors

    ZenPacks.community.HPMon

    ZenPacks.community.VMwareESXMonitor

    ZenPacks.community.VMwareEsx

    ZenPacks.community.WMIDataSource

    ZenPacks.community.WMIPerf_Exchange2003

    ZenPacks.community.WMIPerf_TerminalServer

    ZenPacks.community.deviceAdvDetail

    ZenPacks.vmware.VirtualMachines

    ZenPacks.zenoss.HPMonitor

    ZenPacks.zenoss.HttpMonitor

    ZenPacks.zenoss.ZenossVirtualHostMonitor

  • Alan Milligan Rank: Green Belt
    Alan Milligan
    95 posts since
    Feb 12, 2010
    Currently Being Moderated
    2. Sep 6, 2011 3:40 PM (in response to rphil001)
    Re: Upgraded to 3.2, now broken

    I've seen this issue too.  It's actually related to a bad ZCatalog entry.  Either in Devices/componentSearch or Devices/deviceSearch which have titleOrId indexes or metadata.  Unfortunately, rebuilding these catalogs didn't fix the issue (and a grok of the entire code base didn't highlight any obvious issue - ie any loose context.titleOrId instead of context.titleOrId() ...)

     

    I also converted the calls to simplejson which while not fixing the issue will be considerably faster for python < 2.7.

     

    So, it's nothing to do with ZenPacks .. just Zenoss Core

  • artifact Rank: Green Belt
    artifact
    303 posts since
    Mar 11, 2008
    Currently Being Moderated
    3. Sep 6, 2011 4:41 PM (in response to gadams)
    Re: Upgraded to 3.2, now broken

    I'm seeing the same problem. I'm trying to fix it by clearing the devicesearch and ipsearch and componentsearch catalogs, re-executing zenmigrate manually, and running zencatalog.py --force. I'll let you know how it goes.

  • dhopp Rank: Green Belt
    dhopp
    184 posts since
    Jul 17, 2007
    Currently Being Moderated
    4. Sep 6, 2011 5:13 PM (in response to Alan Milligan)
    Re: Upgraded to 3.2, now broken

    Has a ticket been opened on http://dev.zenoss.org/trac ?

     

    --Dennis

  • rphil001 Rank: White Belt
    rphil001
    7 posts since
    Jul 25, 2011
    Currently Being Moderated
    5. Sep 7, 2011 3:35 AM (in response to artifact)
    Re: Upgraded to 3.2, now broken

    How dod you get on, any luck with fixing it? im in the process of rolling back to 3.1.0 and restoring.

  • artifact Rank: Green Belt
    artifact
    303 posts since
    Mar 11, 2008
    Currently Being Moderated
    6. Sep 7, 2011 2:21 PM (in response to gadams)
    Re: Upgraded to 3.2, now broken

    I've had some luck, everything is working (event console and Infrastructure page) execpt for anything that involves modeling/indexing. So modeling an existing device, adding a new device, etc. is all borken. I'm still getting errors like this anything the new_index_object() function is called:

     

     

    2011-09-07T11:17:13 ERROR zen.Zuul new_index_object() takes exactly 1 argument (2 given)
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/Zuul/routers/device.py", line 242, in moveDevices
    facade.moveDevices(uids, target)
  File "/usr/local/zenoss/zenoss/Products/Zuul/facades/devicefacade.py", line 350, in moveDevices
    dev.setLocation(targetname)
  File "/usr/local/zenoss/zenoss/Products/ZenModel/Device.py", line 1544, in setLocation
    self.index_object()
  File "/usr/local/zenoss/zenoss/Products/ZenModel/Device.py", line 1967, in index_object
    super(Device, self).index_object()
TypeError: new_index_object() takes exactly 1 argument (2 given)

     

    I've seen a couple of other threads like this, but I can't find an applicable bug report on Trac, so I'm opening a report...

  • artifact Rank: Green Belt
    artifact
    303 posts since
    Mar 11, 2008
    Currently Being Moderated
    7. Sep 7, 2011 2:27 PM (in response to artifact)
    Re: Upgraded to 3.2, now broken

    http://dev.zenoss.com/trac/ticket/7862

     

    Please contribute/add to the ticket any pertaining data you guys might have...

  • gadams Rank: White Belt
    gadams
    Currently Being Moderated
    8. Sep 8, 2011 4:12 AM (in response to artifact)
    Re: Upgraded to 3.2, now broken

    Tried to login to Trac with zenoss/zenoss but this doesn't work, instructions say:

    "Login (top right) with zenoss/zenoss or your Trac account (accounts available, contact community@…)"

    community@... ????

  • gadams Rank: White Belt
    gadams
    Currently Being Moderated
    9. Sep 8, 2011 5:54 AM (in response to gadams)
    Re: Upgraded to 3.2, now broken

    Hmm, can login to dev.zenoss.org/trac with IE 8 but not with Firefox 3.6 .  IE it is then ...

  • Rank: Brown Belt
    Nick Yeates
    260 posts since
    Mar 30, 2009
    Currently Being Moderated
    10. Sep 14, 2011 1:25 PM (in response to artifact)
    Re: Upgraded to 3.2, now broken

    You need to remove your old PreUpgrade Zenpack with:

    $ZENHOME/bin/zenpack --remove ZenPacks.zenoss.PreUpgrade30

     

    If this gives an error, look to see if you have this ZenPack via:

    $ZENHOME/bin/zenpack --list

     

    -----

     

    Two patches have been developed that will tell users this, for future versions.

    See http://dev.zenoss.com/trac/ticket/7862

  • gadams Rank: White Belt
    gadams
    Currently Being Moderated
    11. Sep 15, 2011 6:20 AM (in response to Nick Yeates)
    Re: Upgraded to 3.2, now broken

    We have never used pre-3.0 Zenoss so don't have a preupgrade Zenpack:

     

    [zenoss@ug ~]$ $ZENHOME/bin/zenpack --remove ZenPacks.zenoss.PreUpgrade30
    ERROR:zen.ZenPackCmd:zenpack command failed
    Traceback (most recent call last):
      File "/opt/zenoss/Products/ZenUtils/zenpack.py", line 406, in <module>
        zp.run()
      File "/opt/zenoss/Products/ZenUtils/zenpack.py", line 165, in run
        self.options.removePackName)
    ZenPackException: ZenPack ZenPacks.zenoss.PreUpgrade30 is not installed.
    [zenoss@ug ~]$

    [zenoss@ug ~]$ $ZENHOME/bin/zenpack --list
    ZenPacks.HP.Proliant.MIBs (/opt/zenoss/ZenPacks/ZenPacks.HP.Proliant.MIBs-1.0-py2.6.egg)
    ZenPacks.SCC.ShowGraphPortlet (/opt/zenoss/ZenPacks/ZenPacks.SCC.ShowGraphPortlet-1.03-py2.6.egg)
    ZenPacks.community.HPMon (/opt/zenoss/ZenPacks/ZenPacks.community.HPMon-2.2.72.egg)
    ZenPacks.community.deviceAdvDetail (/opt/zenoss/ZenPacks/ZenPacks.community.deviceAdvDetail-2.7.egg)
    ZenPacks.sodonnell.MACReport (/opt/zenoss/ZenPacks/ZenPacks.sodonnell.MACReport-1.0.1-py2.6.egg)
    ZenPacks.zenoss.ApacheMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.ApacheMonitor-2.1.2-py2.6.egg)
    ZenPacks.zenoss.DellMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.DellMonitor-2.1.0-py2.6.egg)
    ZenPacks.zenoss.DigMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.DigMonitor-1.0.2-py2.6.egg)
    ZenPacks.zenoss.DnsMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.DnsMonitor-2.0.2-py2.6.egg)
    ZenPacks.zenoss.EsxTop (/opt/zenoss/ZenPacks/ZenPacks.zenoss.EsxTop-1.0.3-py2.6.egg)
    ZenPacks.zenoss.FtpMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.FtpMonitor-1.0.2-py2.6.egg)
    ZenPacks.zenoss.HPMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.HPMonitor-2.1.0-py2.6.egg)
    ZenPacks.zenoss.HttpMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.HttpMonitor-2.0.3-py2.6.egg)
    ZenPacks.zenoss.IRCDMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.IRCDMonitor-1.0.2-py2.6.egg)
    ZenPacks.zenoss.JabberMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.JabberMonitor-1.0.2-py2.6.egg)
    ZenPacks.zenoss.LDAPMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.LDAPMonitor-1.2.3-py2.6.egg)
    ZenPacks.zenoss.LinuxMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.LinuxMonitor-1.1.5-py2.6.egg)
    ZenPacks.zenoss.MySqlMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.MySqlMonitor-2.1.3-py2.6.egg)
    ZenPacks.zenoss.NNTPMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.NNTPMonitor-1.0.2-py2.6.egg)
    ZenPacks.zenoss.NtpMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.NtpMonitor-2.0.3-py2.6.egg)
    ZenPacks.zenoss.RPCMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.RPCMonitor-1.0.2-py2.6.egg)
    ZenPacks.zenoss.XenMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.XenMonitor-1.0.3-py2.6.egg)
    ZenPacks.zenoss.ZenAWS (/opt/zenoss/ZenPacks/ZenPacks.zenoss.ZenAWS-1.0.3-py2.6.egg)
    ZenPacks.zenoss.ZenJMX (/opt/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.5.3-py2.6.egg)
    ZenPacks.zenoss.ZenossVirtualHostMonitor (/opt/zenoss/ZenPacks/ZenPacks.zenoss.ZenossVirtualHostMonitor-2.3.6-py2.6.egg)
    [zenoss@ug ~]$

     

    
  • gadams Rank: White Belt
    gadams
    Currently Being Moderated
    12. Sep 16, 2011 4:39 AM (in response to gadams)
    Re: Upgraded to 3.2, now broken

    Ticket #7862  says (verification defect: Fixed)

     

    

     

    however the fix (as above) is not applicable to our system.

     

    I am planning to install 3.1 on a test VM and try to replicate the upgrade issue.  We don't want to have our production system down for too long (currently OK back running 3.1).  I did run zenfixit.py on this and it didn't report any errors.

  • gadams Rank: White Belt
    gadams
    Currently Being Moderated
    13. Sep 20, 2011 9:33 AM (in response to gadams)
    Re: Upgraded to 3.2, now broken

    I have made progress with this on a test machine.  I replicated the production environment then upgraded to 3.2.0 .

    I found that running reindex() in zendmd would provoke the problem about the non serializable object.  By hacking some debugging code into /opt/zenoss/Products/Zuul/catalog/global_catalog.py I found the problem was with a cpqSsChassis object from

    /opt/zenoss/ZenPacks/ZenPacks.community.HPMon-2.2.72.egg/ZenPacks/community/HPMon/cpqSsChassis.py .  This defined "name" as a string.

    I hacked the above to remove that definition from the code and also further hacked global_catalog.py as follows:

        def searchExcerpt(self):
            o = self._context
            if type(o.name) == type("a"):
                    print o.name
                    del(o.name)
            return '%s <span style="font-size:smaller">(%s)</span>' % (
                o.name(), o.getParentDeviceName())

    

     

    Now running reindex(0 removes the miscreant from the database.  I shall contact the Zenpack author to get this fixed, or should the code cope with o.name being a string rather than a method?

     

    I also had to run zenmigrate a couple of times.

     

    There must be a better way of removing this attribute from the object, using Zope manager?

More Like This

  • Retrieving data ...

Incoming Links

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points