mray has changed the topic to: Zenoss Developers are here (10:04:37 AM)
10:05:44 AM mray: Another Zenoss IRC session, jplouis will be handling questions
10:06:08 AM rmatte: rocket: long time no see
10:08:16 AM sergeymasushko: hi to Zenoss staff. Could you please take a look at this http://fpaste.org/MIpI/ since upgrade to 3.0.2 I do not receive events from my shell script ran by zencommand. however I can see another events from the zencommand daemon. Could you suggest me what I should check there?
10:11:11 AM rmatte: $laLoadInt1+$laLoadInt15/$laLoadInt5+sqrt(1000)
10:11:13 AM rmatte: oops
10:12:06 AM rocket: hey rmatte
10:16:29 AM jplouis: sergey: so no events show up at all from zencommand?  I don't see any send errors in the log you provided.  Do you have any transforms that may be dropping the events?
10:17:04 AM sergeymasushko: I do not use transforms...
10:19:15 AM sergeymasushko: jplouis: from zencommand I see only events such a "bla-bla command timed out bla-bla" my script exit with status 2 if there is an error but no event appear.
10:19:33 AM sergeymasushko: it works just fine on 2.5.4
10:19:45 AM sergeymasushko: worked*
10:24:42 AM sergeymasushko: also I noticed that zencatalog.log is not updating and if run "zenoss stop" it stay alive and only thing what I can do is "kill -9 <pid>". could it be relevant?
10:25:21 AM sergeymasushko: zenoss was updated from source, OS is freebsd
10:27:54 AM Parabola|W: good morning
10:28:02 AM Parabola|W: And hello developers
10:29:38 AM jplouis: sergey are you on trunk or the 3.0.x branch?
10:30:02 AM sergeymasushko: jplouis: sorry, my fault. there is a transform... http://fpaste.org/DD0n/ but there are no relevant events: http://fpaste.org/DD0n/
10:32:41 AM sergeymasushko: jplouis: 3.0.2
10:32:53 AM jplouis: the debug output you provided shows that the event is being created and queued, So I'm not really sure where it is being dropped. Have you checked the event history?
10:34:15 AM jplouis: zencatalog is a one time script.  It is supposed to be run before upgrading from 2.5.x to 3.0.x.  Once it finishes it should log a success message.
10:34:15 AM st3v3o: Is there a way to remove a transform not using the web UI
10:34:29 AM st3v3o: we entered a bad transform that crashed zenoss
10:34:57 AM rmatte: st3v3o: you can't even access the zope management console?
10:35:07 AM st3v3o: nope dude super borked it
10:35:16 AM rmatte: can you get in to zendmd?
10:35:25 AM rmatte: also, what the heck kind of a transform was it that it caused that?
10:35:56 AM st3v3o: to move windows events to history if the last event time was over an hour
10:36:11 AM st3v3o: but he didn't define all the variables…i think he copied and pasted code together
10:36:12 AM Sam-I-Am: rmatte: is there a way to template which network interfaces zenoss monitors by default?
10:36:28 AM st3v3o: nope I can't even get into zendmd
10:36:31 AM rmatte: Sam-I-Am: what do you mean?
10:36:41 AM rmatte: st3v3o: then you're completely hosed
10:36:43 AM Sam-I-Am: rmatte: by default, it monitors all the interfaces it finds.
10:36:44 AM rmatte: the problem is in zope
10:36:55 AM Sam-I-Am: rmatte: which also makes a ton of rrd files by default
10:36:59 AM rmatte: Sam-I-Am: oh, there's a zProperty to ignore interfaces based on names
10:37:00 AM cgibbons: wow
10:37:01 AM st3v3o: yeah I get this Daemon: zopectl Error: no values for 'path'; 1 required
10:37:02 AM st3v3o: (line 747 in file:///opt/zenoss/etc/zope.conf)
10:37:02 AM st3v3o: For help, use /opt/zenoss/lib/python/Zope2/Startup/zopectl.py -h
10:37:04 AM sergeymasushko: jplouis: OK. I checked event history and didn't find anything there... I removed the event transform, now it works. thanks for the point.
10:37:34 AM sergeymasushko: *confused... *
10:37:58 AM rmatte: It sounds like you might have actually deleted an object path in zope
10:38:06 AM rmatte: can I see the transform code?
10:38:17 AM st3v3o: sure let me find it in the logs
10:38:20 AM rmatte: k
10:39:29 AM rmatte: sergeymasushko: maybe the transform was dropping them rather than moving them to history
10:39:34 AM Sam-I-Am: rmatte: does that take regexes?
10:39:46 AM rmatte: Sam-I-Am: yes
10:39:55 AM Sam-I-Am: cool, got it thx
10:40:06 AM rmatte: the default is a regex
10:40:17 AM rmatte: it's something like ^(name1|name2|name3)
10:40:27 AM rmatte: which tells it to ignore any interfaces that start with those names
10:40:30 AM st3v3o: http://pastebin.com/NZT2QSAZ
10:40:37 AM Sam-I-Am: be even more cool if i could do it by group instead of device
10:40:52 AM st3v3o: well I guess I can alway restore from a zenoss backup
10:40:53 AM sergeymasushko: rmatte: sure, the events were dropped... but I do not understand why...
10:41:01 AM st3v3o: I have one from 10pm last night that is good
10:41:15 AM rmatte: st3v3o: you can do it by device class, but not by ground
10:41:17 AM rmatte: group*
10:41:27 AM rmatte: erm, that was meant for Sam-I-Am
10:41:28 AM rmatte: lol
10:42:18 AM Sam-I-Am: heh
10:42:27 AM rmatte: st3v3o: what version of Zenoss?
10:43:20 AM rmatte: I'm trying to see what this transform could have possibly done to cause that...
10:43:49 AM st3v3o: 3.0.2
10:44:05 AM rmatte: k, just for future reference, dmd is automatically define for transforms
10:44:12 AM st3v3o: what do you mean not by group ?
10:44:22 AM rmatte: st3v3o: that was directed at Sam-I-Am
10:44:29 AM rmatte: I accidentally put your name
10:44:42 AM st3v3o: ok
10:44:53 AM st3v3o: yeah I don't know why he definded dmd again
10:45:17 AM rmatte: wouldn't have hurt it, it's just unncessary
10:45:23 AM rmatte: unnecessary*
10:45:40 AM rmatte: he didn't need to import Globals either
10:45:57 AM st3v3o: yeah just needs time
10:47:04 AM st3v3o: the issue is that he is calling Mins…but didn
10:47:08 AM st3v3o: didn't define it anywhere
10:47:30 AM sergeymasushko: one more thing... I have an event http://fpaste.org/XmTq/ what is wrong with this python expressin?
10:47:32 AM st3v3o: under the Hours variable
10:47:40 AM st3v3o: which still looks unneeded
10:47:51 AM rmatte: the functions he's using...
10:48:20 AM rmatte: he could have just done...
10:48:42 AM rmatte: for e in dmd.ZenEventManager.getEventList(
10:48:44 AM rmatte:     evt = dmd.ZenEventManager.getEventDetailFromStatusOrHistory(e.evid)
10:48:59 AM rmatte: he's doing so many steps just to get that same result
10:49:30 AM st3v3o: alright time to clean this up a little
10:51:29 AM rmatte: sergeymasushko: checking...
10:51:47 AM st3v3o: http://pastebin.com/dTJqJyNL
10:51:56 AM st3v3o: ok a little cleaner I think
10:52:26 AM rmatte: sergeymasushko: that expression looks fine
10:52:45 AM sergeymasushko: rmatte: that is why I am confused...
10:53:06 AM rmatte: sergeymasushko: best guess, bad RRD files
10:53:59 AM rmatte: If it doesn't have usedBlocks to work with it's going to fail
10:54:05 AM rmatte: same with total blocks
10:54:49 AM rmatte: ok, back to st3v3o's transform...
10:55:26 AM rmatte: he did define Minutes
10:55:36 AM rmatte: #
10:55:36 AM rmatte:  31   TimeDiff = CurrTime - epoch
10:55:36 AM rmatte: #
10:55:36 AM rmatte:  32   Minutes  = (TimeDiff / 60)
10:55:48 AM st3v3o: yeah but not Min
10:55:55 AM rmatte: yeh, just noticed that
10:56:00 AM st3v3o: http://fpaste.org/Dm94/
10:56:00 AM rmatte: he abbreviated it on the next line
10:56:01 AM rmatte: lol
10:56:06 AM st3v3o: here is my newest version
10:56:15 AM st3v3o: pastebin stopped working for me all the sudeen
10:56:16 AM rmatte: I'm still trying to figure out how that could have hosed zope
10:57:04 AM st3v3o: evt = dmd.ZenEventManager.getEventDetailFromStatusOrHistory(e.evid)  ….this will also get events already in history ?
10:57:04 AM rmatte: what exactly does this transform do, auto-clear events older than an hour or something?
10:57:20 AM rmatte: yes, hence the name
10:57:24 AM rmatte: from status or history
10:57:25 AM rmatte:
10:57:26 AM rmatte: lol
10:57:30 AM st3v3o: Just making sure
10:57:37 AM rmatte: hehe
10:57:48 AM st3v3o: we only wanna search events currenlty in the event console
10:58:07 AM rmatte: that's what for e in dmd.ZenEventManager.getEventList( does
10:58:15 AM rmatte: it goes through events that are currently in the event console
10:58:28 AM st3v3o: oh failboat
10:58:32 AM rmatte: ?
10:58:40 AM st3v3o: I didn't read above..my bad
10:58:48 AM rmatte: hehe
10:59:01 AM rmatte: can you explain what this is doing?
10:59:08 AM rmatte: clearing events older than an hour?
10:59:12 AM st3v3o: the main goal is to remove windows events that come in from the windows event log that just hang there and never get a clear event
10:59:23 AM rmatte: right
10:59:29 AM rmatte: so what span of time do you want to clear them for?
10:59:43 AM sergeymasushko: rmatte: how can I check that rrd files are bad?
10:59:47 AM st3v3o: if they are older then an hour and the count hasn't incremented
11:00:23 AM rmatte: k, I have a cleaner way to do that...
11:00:33 AM st3v3o: I'm all for that
11:00:55 AM rmatte: one sec
11:01:12 AM Parabola|W: wow
11:02:02 AM rmatte: evt = dmd.ZenEventManager.getEventDetailFromStatusOrHistory(e.evid)
11:02:02 AM rmatte: pattern = '%Y/%m/%d %H:%M:%S'
11:02:02 AM rmatte: epoch = int(time.mktime(time.strptime(evt.lastTime.split('.')[0], pattern)))
11:02:02 AM rmatte: if time.time() - epoch > 3600:
11:02:02 AM rmatte:     dmd.ZenEventManager.manage_deleteEvents(evt.evid)
11:02:06 AM rmatte: that'll do what you want
11:02:14 AM rmatte: I wrote that for my ticket creation daemon
11:02:27 AM rmatte: it also auto-clears events older than an hour since the Zenoss mechanism for doing it is broken
11:02:32 AM OiPolloi: Hello
11:02:38 AM rmatte: hi
11:03:12 AM OiPolloi: anyone has any tips for optimizing ZenOSS? I have a ZenOSS installation but it's getting really slow, specially when I do some operations like modelling a device or seeing the whole Infrastructure tree
11:03:29 AM rmatte: oh, you must have taken that from me at some point
11:03:43 AM rmatte: I didn't even notice you are using almost the same code
11:03:44 AM rmatte: lol
11:03:46 AM nyeates: Oi:gotta find the bottle neck first
11:04:10 AM nyeates: maybe its zenhub, maybe its zope....look at their cpu and mem usage and logs too....figure out where bottleneck is
11:04:13 AM rmatte: st3v3o: but yeh, you don't need to use the variables timediff, minutes, etc... just do it the way I showed
11:04:59 AM st3v3o: http://fpaste.org/3RMo/
11:04:59 AM rmatte: OiPolloi: yes...
11:05:00 AM st3v3o: so that
11:05:20 AM rmatte: OiPolloi: docs/DOC-2521
11:05:52 AM rmatte: st3v3o: exactly, see how simple that is? lol
11:05:53 AM OiPolloi: oh, nice
11:06:01 AM OiPolloi: thanks for the document
11:06:28 AM nyeates: nice doc ryan
11:06:43 AM rmatte: nyeates: I didn't write the whole thing, I just updated it, but thanks
11:06:55 AM OiPolloi: i'm still using 3.0.0. do you think it's worth it upgrading to 3.0.2 (i'm afraid of latent bugs)
11:07:08 AM rmatte: OiPolloi: very worth it, 3.0.2 addresses a lot of bugs
11:07:14 AM mray: 3.0.2 is definitely better than 3.0.0
11:07:16 AM rmatte: and a 3.0.3 release will be coming out
11:07:23 AM OiPolloi: ok, will do that then.
11:07:25 AM nyeates: yeah weve got something similar for customers.... i think we toy with putting it into docs
11:07:41 AM willwh: hi folks
11:08:42 AM OiPolloi: one more thing, what do you consider to be a large zenoss installation? do you think 144 machines is a large installation? it's currently with 4 GB of RAM
11:09:23 AM cgibbons: tiny
11:09:37 AM mray: I've seen smaller
11:09:43 AM OiPolloi: good
11:10:03 AM OiPolloi: i'm using a remote collector, but mostly because of network issues
11:11:13 AM nyeates: 4GB should be good - unles they are swtiches with thousands of datapoints....set mysql settings to use it right
11:12:49 AM OiPolloi: actually, mysql is not using much memory
11:13:00 AM OiPolloi: i'd say mysql is the least of my problems
11:13:33 AM fracBlend: hey
11:13:42 AM fracBlend: File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/zenoss/snmp/InterfaceMap.py", line 135, in process
11:13:42 AM fracBlend: if row['iptype'] != 1:
11:13:42 AM fracBlend: KeyError: 'iptype'
11:13:50 AM fracBlend: any ideas what might cause this?
11:14:57 AM nyeates: Oi: set zope cache-size set to 50000
11:15:15 AM nyeates: or play with the number...restarting zenoss each time
11:15:18 AM nyeates: see how it works
11:15:52 AM Sam-I-Am: nyeates: hey, you're the guy who worked on one of my cases
11:16:25 AM nyeates: This setting will most likely have the single biggest impact on UI performance, it is related to how many objects are in the ZEO database
11:16:35 AM OiPolloi: nyeates: ohh, nice... let me try that
11:16:43 AM Parabola|W: nyeates,  is there an article or white paper on that setting?
11:16:58 AM Parabola|W: i've got a 302 install thats on a nice vmware box, single vcpu with 4gb of ram
11:16:59 AM Parabola|W: and the UI lags
11:17:07 AM Parabola|W: less than 200 objects being monitored
11:17:32 AM nyeates: there is a KB article for customers....see the cool things u get when u sign up!
11:18:01 AM Parabola|W: yeah, but honestly 25 grand is crazy for a product that cannot even monitor dynamic hosts
11:18:04 AM Sam-I-Am: nyeates: i'm the guy who griped about the userland reports not working
11:18:08 AM Parabola|W: should have been like the 3rd thing that was added to zenoss, and it wasnt
11:18:15 AM Parabola|W: not to be rude, i do like the product
11:18:22 AM Parabola|W: but i have to keep a zabbix vm around just for that reason
11:19:39 AM rmatte: Parabola|W: which dynamic components in particular?
11:19:53 AM cgibbons: DHCP hosts?
11:19:56 AM Parabola|W: rmatte,  i have a couple hundred small businesses and retail shops that i'd like to monitor their wan links
11:20:02 AM Parabola|W: and alot of them their only option is att dsl
11:20:06 AM Parabola|W: yes
11:20:19 AM Parabola|W: and for whatever reason, the leases last about 24 hours
11:20:21 AM rmatte: oh, I see, you mean monitor via dns
11:20:23 AM Parabola|W: yes.
11:20:35 AM rmatte: yeh, I agree, that's a really basic feature that should have been implemented
11:20:37 AM Parabola|W: i found a script on the forumsfor it
11:20:45 AM Parabola|W: but if you have any with static IP's it wipes thoes out too
11:20:47 AM Parabola|W: so i cannot use it
11:20:50 AM rmatte: yeh, it would be simple enough to script
11:20:54 AM rmatte: ah
11:21:06 AM Parabola|W: yeah theres no logic to do that apparently
11:21:17 AM Parabola|W: what i do know is i offered to fund the development of that feature
11:21:18 AM rmatte: you'd have to add a zProperty
11:21:21 AM Parabola|W: and no one seemed interested
11:21:26 AM rmatte: and have the script use that to determine if it's static or not
11:21:32 AM rmatte: that's the only way you'd manage it
11:21:36 AM Parabola|W: yeah
11:21:55 AM Parabola|W: again, not trying to insult you devs, i do like the product alot
11:22:06 AM Parabola|W: its leaps and bounds better in many many ways than the compitition
11:22:20 AM rmatte: yeh, it is missing a few little things like that
11:22:33 AM rmatte: also the auto-event clearing not working for Critical events is another
11:22:42 AM rmatte: just little stuff like that which seems to never get fixed
11:22:43 AM rmatte: lol
11:22:48 AM Parabola|W: +1
11:22:56 AM rmatte: but oh well
11:22:59 AM Parabola|W: we killed the conversation
11:23:06 AM Parabola|W: now look what you did
11:23:06 AM rmatte: I just coded my own event clearing in to the ticket creation daemon that I wrote
11:23:10 AM rmatte: so that's working for me for now
11:23:13 AM Parabola|W: sweet
11:23:21 AM Parabola|W: i'm actually looking to see if ipmonitor does WMI
11:23:30 AM Parabola|W: i have used that product in the past, i was a big fan
11:23:50 AM rmatte: well, I know solarwinds does WMI
11:23:54 AM rmatte: but probably not IPMonitor
11:24:01 AM Parabola|W: solarwinds is too pricey
11:24:04 AM Parabola|W: iirc
11:24:08 AM rmatte: solarwinds is a piece of junk
11:24:15 AM rmatte: we have it at work, I absolutely hate it
11:24:29 AM rmatte: the web interface is a nightmare to work with
11:24:34 AM rmatte: no inherited properties
11:24:49 AM rmatte: input fields that you have to change from True to False by typing instead of dropdowns
11:24:53 AM rmatte: just lots of stupidities
11:24:57 AM Parabola|W: lol
11:25:07 AM Parabola|W: hmm
11:25:40 AM rmatte: the only reason we have it is because it's "pretty"
11:26:19 AM Parabola|W: uh oh
11:26:20 AM Parabola|W: Monitor routers, switches, and servers out-of-the-box via SNMP and WMI
11:26:23 AM Parabola|W: ipmonitor does do it
11:26:42 AM Parabola|W: maybe ill get a trial. the last version i used was in 2008 and the install was done all wrong, notifications were bad.e.ct
11:26:47 AM Parabola|W: so i set it up from scratch took me an afternoon
11:26:57 AM rmatte: it runs on windows
11:27:02 AM rmatte: that's another negative
11:27:15 AM rmatte: the words windows and stability don't go together
11:27:26 AM OiPolloi: nyeates: the ui seems a lot snappier
11:27:39 AM rmatte: OiPolloi: yeh, it should be
11:27:39 AM Parabola|W:  rmatte  yeah
11:27:45 AM OiPolloi: i'll take a good look at the other optimizations on that DOC
11:27:48 AM OiPolloi: thanks
11:28:12 AM rmatte: np
11:30:33 AM klinstifen: afternoon all
11:30:40 AM klinstifen: how do you install mibs in 3.0.2?
11:30:51 AM rmatte: the same way as in 2.5
11:30:53 AM klinstifen: i don't see an option in the gui to loab a MIB file
11:30:54 AM rmatte: and 2.4
11:30:56 AM rmatte: and 2.3
11:30:56 AM rmatte: lol
11:31:05 AM rmatte: klinstifen: just use zenmib from the commandline
11:31:09 AM rmatte: don't bother with the UI
11:31:20 AM klinstifen: so uhh not the same then
11:31:36 AM rmatte: I'm sure there's a way to do it from the UI too
11:31:45 AM rmatte: but the commandline is the best way to do it
11:31:49 AM rmatte: always has been
11:32:01 AM rmatte: trust me, if I had built the Cisco Mibs pack via the UI I would have shot myself
11:32:21 AM rmatte: just put the Mibs in a directory on the server, become the Zenoss user, and do "zenmib run *" inside the directory
11:33:22 AM klinstifen: thx
11:34:31 AM klinstifen: and to refresh my memory, after the mibs are loaded zenoss will start interpreting oid's correctly, right?
11:34:52 AM rmatte: what I've noticed is that after installing new Mibs you have to restart Zenoss for them to take effect
11:35:05 AM rmatte: and then yes, provided you have the correct Mibs installed the traps will be translated
11:35:45 AM Parabola|W: lo
11:35:50 AM Parabola|W: i changed the zope cache
11:35:53 AM Parabola|W: zope no longer starts
11:36:05 AM rmatte: Parabola|W: what did you change it to?
11:36:15 AM Parabola|W: 25k
11:36:16 AM rmatte: it can't be something ridiculously high or it'll exceed your memory space
11:36:54 AM Parabola|W: did he not say set it to 50k?
11:37:02 AM Parabola|W:  i believe i see that in the buffer a couple min ago
11:37:09 AM fracBlend: Anyone know how to get juniper ssg interfaces to model?
11:37:10 AM rmatte: I have mine set as 60k on one of my servers
11:37:13 AM rmatte: so yeh, 25k is fine
11:37:17 AM Parabola|W:
11:37:21 AM rmatte: what are you seeing in zope.log?
11:38:39 AM rmatte: I typed the "it can't be something ridiculously high" before I read your "25k"
11:39:06 AM OiPolloi: i just set mine to 50k and it's working fine, apparently
11:39:49 AM OiPolloi: one more thing: each time i restart zenoss, and sometimes when i'm doing something intensive, i get a few heartbeat events
11:39:55 AM OiPolloi: is this normal?
11:40:05 AM rmatte: OiPolloi: ignore those
11:40:13 AM rmatte: They are basically useless since they are overly-sensitive
11:40:20 AM rmatte: that's something that needs to be fixed
11:40:49 AM OiPolloi: yeah, i thought it was something like that... because everything seems to be working fine, even when I get those
11:40:58 AM rmatte: yeh, they won't affect anything