[2009-05-14 09::59:39] mrayzenoss: Good morning/afternoon/evening everybody. mrchippy and I will be here today, perhaps more Zenossians will drop in as well
[2009-05-14 10::02:57] __ian__: Wow, Zenoss dev dept nearly 100% represented here.
[2009-05-14 10::03:48] mrayzenoss: except the guy who's really supposed to be here
[2009-05-14 10::04:10] __ian__: Throw something at him.
[2009-05-14 10::04:53] burlyscudd: mrayzenoss: yo
[2009-05-14 10::04:56] mrchippy: don't know where he is. he's off getting his pituitary injections.
[2009-05-14 10::05:24] mrayzenoss: so far it's just Zenoss folks talking to Zenoss folks... so how's Canada Ian?
[2009-05-14 10::05:28] sergeymasushko: I tried to install a dig monitor zenpack... binded template to a device, but when I tried to open the template I got an error, someone tried this zenpack on 2.4?
[2009-05-14 10::07:11] __ian__: sergeymasushko: What's the error?
[2009-05-14 10::08:42] ** JKilgore joined the chat room.
[2009-05-14 10::09:45] sergeymasushko: __ian__: http://pastebin.com/d4d63a3cf
[2009-05-14 10::10:10] jb: hello
[2009-05-14 10::10:33] kalahari8751: I need some guidance on if we are overloading zencommand. We have a schedule with 1166 commands. Recently we've been getting large gaps in our perf graphs. Running the individual templates with zencommand run -v10 -d always succeeds, but I have devices right now where the graphs are very spotty.
[2009-05-14 10::12:43] __ian__: sergeymasushko: Did you restart Zope after installing the pack?
[2009-05-14 10::13:19] sergeymasushko: no
[2009-05-14 10::13:26] __ian__: give that a try.
[2009-05-14 10::15:16] cgibbons: when in doubt
[2009-05-14 10::15:59] __ian__: Well, when you get broken ZenPack errors of any kind, good bet it's because Zope needs a restart to pick up the new code.
[2009-05-14 10::16:35] ** Mosburn joined the chat room.
[2009-05-14 10::17:27] rocket: Hey Guys
[2009-05-14 10::18:42] ** bedwards joined the chat room.
[2009-05-14 10::18:53] ** jcape joined the chat room.
[2009-05-14 10::19:43] mrayzenoss: woohoo, all 5 developers are here
[2009-05-14 10::20:09] rocket: damn and I dont really have any questions for them at this point
[2009-05-14 10::20:10] mrayzenoss: 2.4.1 is slowly creeping out
[2009-05-14 10::20:17] jb: bleh
[2009-05-14 10::20:19] bedwards: i'm here now. presentation to sales ran long.
[2009-05-14 10::20:21] jb: internet problems
[2009-05-14 10::20:46] mrchippy: aha
[2009-05-14 10::22:10] __ian__: Yeah, google is hurting at the moment. Twitter's aflame with reports.
[2009-05-14 10::22:20] jb: yeah its probably not google
[2009-05-14 10::26:08] kalahari8751: Sorry if poor etiquette--anyone see my question about zencommand?
[2009-05-14 10::26:37] mrayzenoss: kalahari8751: go ahead and repost so bedwards can take a look
[2009-05-14 10::26:44] kalahari8751: I need some guidance on if we are overloading zencommand. We have a schedule with 1166 commands. Recently we've been getting large gaps in our perf graphs. Running the individual templates with zencommand run -v10 -d always succeeds, but I have devices right now where the graphs are very spotty.
[2009-05-14 10::27:13] rocket: kalahari8751: what is the load on the system? eg is the processor busy etc?
[2009-05-14 10::27:26] kalahari8751: Ran an strace on zencommand and it seems to be working. Don't know how to diagnose. 4 proc machine, load numbers average about 1.7.
[2009-05-14 10::28:24] rocket: seems reasonable
[2009-05-14 10::28:29] kalahari8751: Yeah
[2009-05-14 10::29:17] rocket: kalahari8751: have you run zencommand stop;zencommand start -v 10 and then looked at the zencommand log?
[2009-05-14 10::29:34] sergeymasushko: how to allow zenoss' users logint to zenoss.bla-bla-bla.net/manage ?
[2009-05-14 10::29:50] kalahari8751: rocket: no, can do though
[2009-05-14 10::30:25] rocket: kalahari8751: that will give verbose output when running against all your commands at once .. might find something in the logs then
[2009-05-14 10::30:48] bedwards: are there any errors in zencommand.log?
[2009-05-14 10::31:55] kalahari8751: bedwards: No, log looks completely normal. Mostly filled with "schedule has x commands" messages.
[2009-05-14 10::32:47] rocket: kalahari8751: normal even with the -v10 logging?
[2009-05-14 10::33:35] kalahari8751: rocket: just restarted zencommand; output is flying by w/results from commands
[2009-05-14 10::33:45] ** chudler joined the chat room.
[2009-05-14 10::34:00] kalahari8751: Seeing success msgs, lots of rrd storing
[2009-05-14 10::36:00] rocket: kalahari8751: what type of commands are you running?
[2009-05-14 10::37:04] kalahari8751: rocket: no, they nearly always work unless there's a known condition (system down, WMI problem, etc.). Most are wmic calls or remote winexe exec calls.
[2009-05-14 10::38:01] rocket: has the zenoss machine been restarted lately?
[2009-05-14 10::38:29] kalahari8751: 11:38:20 up 7 days
[2009-05-14 10::38:44] rocket: and the graphs have been goofy all 7 days?
[2009-05-14 10::39:14] mrayzenoss: 2.4.1 is now out
[2009-05-14 10::40:40] kalahari8751: rocket: We're not seeing network problems. Like I said, zencommand run -v10 -d device always works, but I have graphs that look very gappy. The reason I think it might be load on zencommand is because: last week manager came in looking for some graphs and we saw that they were mostly not painting.
[2009-05-14 10::41:30] rocket: kalahari8751: sorry I am just trying to make sure I have asked everything I would normally ask when troubleshooting ..
[2009-05-14 10::42:01] kalahari8751: rocket: np. Also, I had a device class w/3 perf templates bound, that was working fine. I edited one template and added one new command datasource and two of the perf templates just stopped producing graphs.
[2009-05-14 10::44:17] rocket: any spaces etc in the datasource items .. eg .. I found in my troubleshooting I had a space in the datasource .. but zencommand didnt have the space
[2009-05-14 10::44:49] kalahari8751: rocket: zenoss version is 2.3.3. Tried a 2.4 upgrade but had to revert, that issue is posted in forums. These problems predated the upgrade attempt however. Yes--there are datasources with spaces.
[2009-05-14 10::44:57] rocket: kalahari8751: my zencommands collected data but then never updated the rrd file in that case
[2009-05-14 10::45:29] kalahari8751: rocket: Ahhhh, that may be it. Crap, there's no rename function that doesn't break graphs.
[2009-05-14 10::45:35] rocket: just make sure the zencommand output matches the datasource definition of the name
[2009-05-14 10::47:58] mrayzenoss: yeah, everyone must be sad about the Google outage
[2009-05-14 10::48:00] kalahari8751: rocket: well, it seems to be respecting the spaces: DEBUG:zen.zencommand:Queueing event {'manager': 'localhost', 'eventKey': 'CitrixAppLaunch', 'device': 'devicename', 'eventClass': '/Cmd/Fail', 'summary': 'OK', 'component': 'Citrix App Launch', 'agent': 'zencommand', 'severity': 0}
[2009-05-14 10::48:22] perr0: is it really an outage?
[2009-05-14 10::48:40] rocket: kalahari8751: my issue was something like Citrix App Launch_ connectionTime
[2009-05-14 10::48:41] mrayzenoss: dunno, I'm sure we'll hear something about it once it clears up
[2009-05-14 10::49:00] rocket: kalahari8751: eg there was a " connectionTime" defined in my data source
[2009-05-14 10::49:00] kalahari8751: rocket: OK, so the problem may be the space in the data source name?
[2009-05-14 10::49:18] rocket: kalahari8751: but that doesnt seem to be your issue
[2009-05-14 10::49:22] kalahari8751: rocket: I don't have that issue
[2009-05-14 10::49:49] rocket: kalahari8751: I would wait and see if it keeps happening with the extended output
[2009-05-14 10::49:52] kalahari8751: rocket: I don't have any datapoints with spaces, just datasources
[2009-05-14 10::50:03] ** edwardam_ is now known as edwardam.
[2009-05-14 10::50:22] rocket: ok I have a question for the devs
[2009-05-14 10::50:37] kalahari8751: OK. Back to the original question, is there any guidance on how many commands is too many? Does zencommand ever silently drop items if it gets under too much load?
[2009-05-14 10::50:41] rocket: say I am creating a device that references another device
[2009-05-14 10::51:30] mrayzenoss: rocket: yeah, that's a common use case for zLinks
[2009-05-14 10::51:33] jb: yeah
[2009-05-14 11::21:41] mrayzenoss: Well, it's been a quiet session, but I'll be here as I usually am
[2009-05-14 10::02:57] __ian__: Wow, Zenoss dev dept nearly 100% represented here.
[2009-05-14 10::03:48] mrayzenoss: except the guy who's really supposed to be here
[2009-05-14 10::04:10] __ian__: Throw something at him.
[2009-05-14 10::04:53] burlyscudd: mrayzenoss: yo
[2009-05-14 10::04:56] mrchippy: don't know where he is. he's off getting his pituitary injections.
[2009-05-14 10::05:24] mrayzenoss: so far it's just Zenoss folks talking to Zenoss folks... so how's Canada Ian?
[2009-05-14 10::05:28] sergeymasushko: I tried to install a dig monitor zenpack... binded template to a device, but when I tried to open the template I got an error, someone tried this zenpack on 2.4?
[2009-05-14 10::07:11] __ian__: sergeymasushko: What's the error?
[2009-05-14 10::08:42] ** JKilgore joined the chat room.
[2009-05-14 10::09:45] sergeymasushko: __ian__: http://pastebin.com/d4d63a3cf
[2009-05-14 10::10:10] jb: hello
[2009-05-14 10::10:33] kalahari8751: I need some guidance on if we are overloading zencommand. We have a schedule with 1166 commands. Recently we've been getting large gaps in our perf graphs. Running the individual templates with zencommand run -v10 -d always succeeds, but I have devices right now where the graphs are very spotty.
[2009-05-14 10::12:43] __ian__: sergeymasushko: Did you restart Zope after installing the pack?
[2009-05-14 10::13:19] sergeymasushko: no
[2009-05-14 10::13:26] __ian__: give that a try.
[2009-05-14 10::15:16] cgibbons: when in doubt
[2009-05-14 10::15:59] __ian__: Well, when you get broken ZenPack errors of any kind, good bet it's because Zope needs a restart to pick up the new code.
[2009-05-14 10::16:35] ** Mosburn joined the chat room.
[2009-05-14 10::17:27] rocket: Hey Guys
[2009-05-14 10::18:42] ** bedwards joined the chat room.
[2009-05-14 10::18:53] ** jcape joined the chat room.
[2009-05-14 10::19:43] mrayzenoss: woohoo, all 5 developers are here
[2009-05-14 10::20:09] rocket: damn and I dont really have any questions for them at this point
[2009-05-14 10::20:10] mrayzenoss: 2.4.1 is slowly creeping out
[2009-05-14 10::20:17] jb: bleh
[2009-05-14 10::20:19] bedwards: i'm here now. presentation to sales ran long.
[2009-05-14 10::20:21] jb: internet problems
[2009-05-14 10::20:46] mrchippy: aha
[2009-05-14 10::22:10] __ian__: Yeah, google is hurting at the moment. Twitter's aflame with reports.
[2009-05-14 10::22:20] jb: yeah its probably not google
[2009-05-14 10::26:08] kalahari8751: Sorry if poor etiquette--anyone see my question about zencommand?
[2009-05-14 10::26:37] mrayzenoss: kalahari8751: go ahead and repost so bedwards can take a look
[2009-05-14 10::26:44] kalahari8751: I need some guidance on if we are overloading zencommand. We have a schedule with 1166 commands. Recently we've been getting large gaps in our perf graphs. Running the individual templates with zencommand run -v10 -d always succeeds, but I have devices right now where the graphs are very spotty.
[2009-05-14 10::27:13] rocket: kalahari8751: what is the load on the system? eg is the processor busy etc?
[2009-05-14 10::27:26] kalahari8751: Ran an strace on zencommand and it seems to be working. Don't know how to diagnose. 4 proc machine, load numbers average about 1.7.
[2009-05-14 10::28:24] rocket: seems reasonable
[2009-05-14 10::28:29] kalahari8751: Yeah
[2009-05-14 10::29:17] rocket: kalahari8751: have you run zencommand stop;zencommand start -v 10 and then looked at the zencommand log?
[2009-05-14 10::29:34] sergeymasushko: how to allow zenoss' users logint to zenoss.bla-bla-bla.net/manage ?
[2009-05-14 10::29:50] kalahari8751: rocket: no, can do though
[2009-05-14 10::30:25] rocket: kalahari8751: that will give verbose output when running against all your commands at once .. might find something in the logs then
[2009-05-14 10::30:48] bedwards: are there any errors in zencommand.log?
[2009-05-14 10::31:55] kalahari8751: bedwards: No, log looks completely normal. Mostly filled with "schedule has x commands" messages.
[2009-05-14 10::32:47] rocket: kalahari8751: normal even with the -v10 logging?
[2009-05-14 10::33:35] kalahari8751: rocket: just restarted zencommand; output is flying by w/results from commands
[2009-05-14 10::33:45] ** chudler joined the chat room.
[2009-05-14 10::34:00] kalahari8751: Seeing success msgs, lots of rrd storing
[2009-05-14 10::36:00] rocket: kalahari8751: what type of commands are you running?
[2009-05-14 10::37:04] kalahari8751: rocket: no, they nearly always work unless there's a known condition (system down, WMI problem, etc.). Most are wmic calls or remote winexe exec calls.
[2009-05-14 10::38:01] rocket: has the zenoss machine been restarted lately?
[2009-05-14 10::38:29] kalahari8751: 11:38:20 up 7 days
[2009-05-14 10::38:44] rocket: and the graphs have been goofy all 7 days?
[2009-05-14 10::39:14] mrayzenoss: 2.4.1 is now out
[2009-05-14 10::40:40] kalahari8751: rocket: We're not seeing network problems. Like I said, zencommand run -v10 -d device always works, but I have graphs that look very gappy. The reason I think it might be load on zencommand is because: last week manager came in looking for some graphs and we saw that they were mostly not painting.
[2009-05-14 10::41:30] rocket: kalahari8751: sorry I am just trying to make sure I have asked everything I would normally ask when troubleshooting ..
[2009-05-14 10::42:01] kalahari8751: rocket: np. Also, I had a device class w/3 perf templates bound, that was working fine. I edited one template and added one new command datasource and two of the perf templates just stopped producing graphs.
[2009-05-14 10::44:17] rocket: any spaces etc in the datasource items .. eg .. I found in my troubleshooting I had a space in the datasource .. but zencommand didnt have the space
[2009-05-14 10::44:49] kalahari8751: rocket: zenoss version is 2.3.3. Tried a 2.4 upgrade but had to revert, that issue is posted in forums. These problems predated the upgrade attempt however. Yes--there are datasources with spaces.
[2009-05-14 10::44:57] rocket: kalahari8751: my zencommands collected data but then never updated the rrd file in that case
[2009-05-14 10::45:29] kalahari8751: rocket: Ahhhh, that may be it. Crap, there's no rename function that doesn't break graphs.
[2009-05-14 10::45:35] rocket: just make sure the zencommand output matches the datasource definition of the name
[2009-05-14 10::47:58] mrayzenoss: yeah, everyone must be sad about the Google outage
[2009-05-14 10::48:00] kalahari8751: rocket: well, it seems to be respecting the spaces: DEBUG:zen.zencommand:Queueing event {'manager': 'localhost', 'eventKey': 'CitrixAppLaunch', 'device': 'devicename', 'eventClass': '/Cmd/Fail', 'summary': 'OK', 'component': 'Citrix App Launch', 'agent': 'zencommand', 'severity': 0}
[2009-05-14 10::48:22] perr0: is it really an outage?
[2009-05-14 10::48:40] rocket: kalahari8751: my issue was something like Citrix App Launch_ connectionTime
[2009-05-14 10::48:41] mrayzenoss: dunno, I'm sure we'll hear something about it once it clears up
[2009-05-14 10::49:00] rocket: kalahari8751: eg there was a " connectionTime" defined in my data source
[2009-05-14 10::49:00] kalahari8751: rocket: OK, so the problem may be the space in the data source name?
[2009-05-14 10::49:18] rocket: kalahari8751: but that doesnt seem to be your issue
[2009-05-14 10::49:22] kalahari8751: rocket: I don't have that issue
[2009-05-14 10::49:49] rocket: kalahari8751: I would wait and see if it keeps happening with the extended output
[2009-05-14 10::49:52] kalahari8751: rocket: I don't have any datapoints with spaces, just datasources
[2009-05-14 10::50:03] ** edwardam_ is now known as edwardam.
[2009-05-14 10::50:22] rocket: ok I have a question for the devs
[2009-05-14 10::50:37] kalahari8751: OK. Back to the original question, is there any guidance on how many commands is too many? Does zencommand ever silently drop items if it gets under too much load?
[2009-05-14 10::50:41] rocket: say I am creating a device that references another device
[2009-05-14 10::51:30] mrayzenoss: rocket: yeah, that's a common use case for zLinks
[2009-05-14 10::51:33] jb: yeah
[2009-05-14 11::21:41] mrayzenoss: Well, it's been a quiet session, but I'll be here as I usually am
