[27-May-2010 10:00:32] <mray> There are a couple of Zenoss developers here today, nzook is on the hook, but he's pretty new, so ask away

[27-May-2010 10:05:34] <Filbert> I'm seeing an issue where various component services are failing to start

[27-May-2010 10:05:44] <Filbert> they're failing with errors that seem to reference non-existant devices

[27-May-2010 10:06:13] <Filbert> I've tried a reindex, increased the zenhub connection timeouts for the most frequent offenders (I was getting timeout issues first)

[27-May-2010 10:06:23] <Filbert> but services still seem to fail with a big traceback in the logs

[27-May-2010 10:06:31] <Filbert> anyone got any advice on how I can go about debugging this further?

[27-May-2010 10:09:39] <mray> could you post the traceback on pastebin.org?

[27-May-2010 10:09:46] <Filbert> sure,one mo

[27-May-2010 10:10:27] <Filbert> http://pastebin.org/285881

[27-May-2010 10:10:53] <Filbert> ldnbh1gxendb02 is a decomissioned server

[27-May-2010 10:11:05] <Filbert> so should have been removed

[27-May-2010 10:11:09] <Filbert> I also can't find it anywhere in the webui

[27-May-2010 10:11:34] <chemist> tips for speeding up web ui anyone?

[27-May-2010 10:12:18] <rocket> chemist: what zenoss version?

[27-May-2010 10:12:35] <chemist> rocket: 2.5.2

[27-May-2010 10:13:05] <rocket> there are a few performance patches you should apply

[27-May-2010 10:13:15] pmcguire is now known as ptmcg

[27-May-2010 10:13:47] <mray> chemist: this seems to have a few good tips: docs/DOC-2521

[27-May-2010 10:14:14] <nzook> Filbert:  Have you looked for the server in dmd?

[27-May-2010 10:14:44] <rocket> @chemist: zenpatch 18589;zenpatch 18890;zenpatch 18908;zenpatch 18941

[27-May-2010 10:15:26] <Filbert> nzook: nope, is there a useful guide on how to do that?

[27-May-2010 10:15:49] <rocket> These patches were found with the following trac query.

[27-May-2010 10:15:51] <rocket> http://dev.zenoss.com/trac/query?status=closed&status=verification&group=patch_state&patch=2.5.3&col=id&col=summary&col=status&col=owner&col=priority&col=milestone&col=component&col=changetime&report=6&order=status

[27-May-2010 10:16:17] fOrsberg is now known as forsberg

[27-May-2010 10:18:00] <nzook> Filbert:  from the command line, type "zendmd".  This will get you a command prompt (it may take a bit).  Do find("ldnbh1gxendb02").

[27-May-2010 10:18:39] <Filbert> nothing returned...

[27-May-2010 10:19:14] <rocket> are you sure that was the managed name?

[27-May-2010 10:19:36] <Filbert> nope :(

[27-May-2010 10:19:44] <Filbert> it's not my original installation I'm afraid :(

[27-May-2010 10:19:49] <Filbert> lemme try by it's IP

[27-May-2010 10:19:54] * chemist applies patches

[27-May-2010 10:20:11] <Filbert> nothing returned for the IP either

[27-May-2010 10:20:23] <Filbert> or FQDN

[27-May-2010 10:20:28] <Filbert> it'd have been one of those

[27-May-2010 10:20:29] <rocket> Filbert: did you run zencheckrels to make sure your relationships are fixed?

[27-May-2010 10:20:40] <Filbert> nope

[27-May-2010 10:20:56] <Filbert> how do I do that?

[27-May-2010 10:21:09] <Filbert> oh, its' a comment

[27-May-2010 10:21:11] <Filbert> er, command

[27-May-2010 10:21:13] <rocket> Filbert: also make sure you are running 2.5.2 if you haven't upgraded .. earlier versions have problems with broken relationships

[27-May-2010 10:21:21] <Filbert> ah, we're on 2.5.1

[27-May-2010 10:21:28] <rocket> get to 2.5.2 asap

[27-May-2010 10:21:38] <rocket> 2.5.1 will eat your lunch....

[27-May-2010 10:21:52] <Filbert> lol

[27-May-2010 10:22:02] <Filbert> ok, was planning ona n upgrade earlier

[27-May-2010 10:22:14] <Filbert> but everything is such a mess here, I was hoping I could put it off for a few weeks ;)

[27-May-2010 10:22:50] <Filbert> zenchkrels running now

[27-May-2010 10:24:50] <Filbert> it seems to be fixing some stuff

[27-May-2010 10:31:42] * Filbert taps his fingers

[27-May-2010 10:31:47] <Filbert> not a very quick process... :/

[27-May-2010 10:31:53] <Filbert> although that might be the size of our ZenOss install

[27-May-2010 10:39:49] <rocket> unfortunately it may crap out if other things access the db at the same time

[27-May-2010 10:42:58] <Simon4> it only commits at the end?

[27-May-2010 10:43:21] <Simon4> or does it commit after each rel fix?

[27-May-2010 10:44:09] <rocket> cant remember .. looking...

[27-May-2010 10:45:49] <rocket> just at the end .. but it could be changed pretty easily

[27-May-2010 10:46:01] <rocket> Products/ZenUtils/CheckRelations.py

[27-May-2010 10:46:23] <Simon4> cool

[27-May-2010 10:46:48] <rocket> it does set savepoints though .. so it should resume ..

[27-May-2010 10:47:16] <rocket> Ideally you do this on a system where nothing is happening though .. no modelling all other daemons are down etc

[27-May-2010 10:47:25] <rocket> but it can take a long time

[27-May-2010 10:47:46] <Simon4> yeah, considering doing it on a dedicated spare box

[27-May-2010 10:47:57] <Simon4> jsut after zeoctl start

[27-May-2010 10:48:02] <mray> rocket: is zenchkrels documented anywhere?

[27-May-2010 10:48:32] <rocket> mray: I am not sure if it is .. ( from our perspective we dont use it too much as it is pretty slow )

[27-May-2010 10:48:58] <rocket> mray: but it is important to anyone who was running 2.5.1 to run it to fix their corrupt zodb

[27-May-2010 10:48:59] <mray> rocket: I'm gonna open a ticket for it

[27-May-2010 10:49:16] <Simon4> rocket: so suggestion is, run zenchkrels, then upgrade to 2.5.2?

[27-May-2010 10:49:20] <rocket> mray: we need to improve it ..

[27-May-2010 10:49:24] <rocket> Simon4: yes

[27-May-2010 10:49:27] <Simon4> before restarting any other daemons etc

[27-May-2010 10:49:33] <Simon4> awesome, will try early next week

[27-May-2010 10:49:43] * Simon4 would love to not have to repair relations every second breath

[27-May-2010 10:49:45] <mray> rocket: I didn't even know what it did

[27-May-2010 10:50:04] <rocket> mray: it checks zope relations and tries to repair them

[27-May-2010 10:50:31] <rocket> mray: eg there is a flaw in 2.5.1 where half of the relation could be deleted and the other half exists

[27-May-2010 10:50:45] <rocket> in certain instances

[27-May-2010 10:50:56] <rocket> specifically if making changes from zendmd in 2.5.1

[27-May-2010 10:51:29] <rocket> then when you try and follow certain relationships and it crosschecks you get tracebacks etc

[27-May-2010 10:52:00] <rocket> internally for enterprise customers we are a bit more surgical about fixing these relationships and have an adhoc tool to do so

[27-May-2010 10:52:28] <rocket> but we *always* recommend getting off 2.5.0/1 and moving to 2.5.2

[27-May-2010 10:52:43] <Filbert> looks like I know what I'm doing tomorrow then :)

[27-May-2010 10:53:24] <mray> rocket: feel free to spice up http://dev.zenoss.org/trac/ticket/6809

[27-May-2010 10:53:27] <rocket> 2.5.2 specifically fixes the issues as far as we know that the zodb can get into an inconsistent state

[27-May-2010 10:58:36] <theacolyte> Oh, hello

[27-May-2010 10:59:14] <theacolyte> Are there any plans of building WMI into core?

[27-May-2010 10:59:15] * Simon4 plots a 2.5.2 upgrade sooner than going to enterprise after reading those comments

[27-May-2010 11:00:18] <rocket> Simon4: have a support contract?

[27-May-2010 11:00:29] <mray> theacolyte: WMI is already in Core, that's how Windows event log and services are monitored

[27-May-2010 11:00:33] <Simon4> rocket: oui

[27-May-2010 11:00:42] <mray> theacolyte: or do you mean the WMI Data Source?

[27-May-2010 11:00:57] <Simon4> yes, even

[27-May-2010 11:01:19] <rocket> Simon4: let me look for a KB article for a tool that you should run instead

[27-May-2010 11:01:24] <Simon4> rocket: thanks

[27-May-2010 11:01:25] <theacolyte> mray: Ah, I thought event logs were provided by egor's plugin

[27-May-2010 11:01:38] <theacolyte> But regardless, yes the WMI datasource

[27-May-2010 11:01:53] <mray> theacolyte: nothing solid yet, but there have been discussions

[27-May-2010 11:02:13] <rocket> Simon4: "Using ZenFixIt to Correct Common Zenoss Issues"

[27-May-2010 11:02:40] * Simon4 goes hunting

[27-May-2010 11:02:43] <theacolyte> Hyperic really did it right for WMI I think - the counters are all in there, you just select them

[27-May-2010 11:02:45] <rocket> Simon4: thats a more surgical repair .. but may not catch everything

[27-May-2010 11:02:51] <theacolyte> No need to look up classes to remember where it's located

[27-May-2010 11:03:01] <Simon4> k, i"ll try it out on a copy of the db early next week

[27-May-2010 11:03:05] <rocket> zenchkrels is really intended to be the broad stroke brush

[27-May-2010 11:04:52] <rocket> mray: added some notes to that ticket for you

[27-May-2010 11:10:40] <mray> New FDS 389 LDAP Server ZenPack: docs/DOC-5889