Transform to clear info events, coding around ConflictErrors, extending OSProcess, zenbatchload.py use, collector over ssl, output status/uptime of all devices

[21-Jul-2011 10:54:55] <Evolution> is there a way to clear out informational alerts for a host via REST, python xmlrpc hook, etc?
[21-Jul-2011 10:55:32] <Evolution> I'm adding the machines via python, but I need them to be green immediately after..
[21-Jul-2011 10:57:00] <nyeates> Evolution: i am looking here: community/documentation/official_documentation/api
[21-Jul-2011 10:57:48] <Evolution> nyeates: yeah... I was afraid of that....
[21-Jul-2011 10:57:57] <Simon4> Evolution: the alert being the "discovered IP xxxx for device Y" one?
[21-Jul-2011 10:58:14] <Evolution> I was hoping for something spelled out in crayon like docs/DOC-2353 :-P
[21-Jul-2011 10:58:41] <Evolution> Simon4: yep.
[21-Jul-2011 10:58:42] <Evolution> that one.
[21-Jul-2011 10:58:44] <Evolution> exactly.
[21-Jul-2011 10:59:12] <Simon4> you could event transform it into the ether
[21-Jul-2011 10:59:15] <JohnnyNOC> maybe a transform to drop them?
[21-Jul-2011 10:59:21] <Simon4> if you don't care about that event at all
[21-Jul-2011 10:59:33] <Evolution> no, I don't, and how would I do this?
[21-Jul-2011 10:59:45] <Evolution> I've been using zenoss for about 3 days now :-P
[21-Jul-2011 10:59:53] * Simon4 has a quick look to try and find one
[21-Jul-2011 11:00:22] <klone> you can delete events via REST, but if i've only ever done it by event id ... not event severity
[21-Jul-2011 11:00:45] <Evolution> my end goal is to be able to automate the deployment of the zenoss server, and automagically add the clients to be monitored from our db.
[21-Jul-2011 11:01:15] <Evolution> basically we're looking to ship a server room in a box with 0 interaction from the end user/admin other than 'yep... that light is green'
[21-Jul-2011 11:01:48] <dpetzel> so is the issue that the server will show a blue icon rather than green icon upon adding it?
[21-Jul-2011 11:01:50] <Simon4> Evolution: just adding a device, please hold
[21-Jul-2011 11:02:02] <Evolution> dpetzel: yep.
[21-Jul-2011 11:02:05] <simonjj> Evolution: info severity events shouldn't keep things from being green
[21-Jul-2011 11:02:49] <Evolution> simonjj: they do though. they show up blue/info in the infra/devices list
[21-Jul-2011 11:02:59] <dpetzel> I've seen that as well
[21-Jul-2011 11:03:07] <iancmcc> Evolution: what version?
[21-Jul-2011 11:03:08] <simonjj> I thought you were talking about overall status
[21-Jul-2011 11:03:14] <Evolution> 3.1.0
[21-Jul-2011 11:03:32] <simonjj> do you ever care about info level alerts ?
[21-Jul-2011 11:03:32] <iancmcc> There is certainly a JSON API call that will close all events for a given device. One sec, I'll suss it out.
[21-Jul-2011 11:03:46] <dpetzel> looking at the JSON API this is very doable, but I dont have any cut/paste code to do it
[21-Jul-2011 11:03:58] <nyeates> Everyone, welcome to our bi-weekly thur dev chat. Only announcements are that we are working towards getting the community on track by opening up access to community.zenoss.org, improving documentation, prepping our 3.2 release, and many other things.
[21-Jul-2011 11:04:00] <Evolution> I don't believe I would care about info level at all, no.
[21-Jul-2011 11:04:17] <iancmcc> The easiest way to figure it out, btw, is just to go do it in the UI and watch the request go out in Firebug.
[21-Jul-2011 11:04:20] <Evolution> this can hold until after the dev chat.
[21-Jul-2011 11:04:36] <Evolution> no need to get in your way
[21-Jul-2011 11:04:37] <simonjj> just drop them then
[21-Jul-2011 11:07:39] <nyeates> Send up any dev or advanced questions - we have some smart people in the room
[21-Jul-2011 11:08:13] <Simon4> Evolution: you want to add an event transform under the "/Status/Snmp" event class that looks something like http://pastie.org/2248915
[21-Jul-2011 11:08:27] <johnf1911> I'm going to run a big import job
[21-Jul-2011 11:08:32] <dpetzel> I'd love to get some developer feedbackon the best direction on this: message/58629#58629
[21-Jul-2011 11:08:34] <Simon4> I highly recommend finding the section in the admin guide that talks about this to find where that is, since getting your head around it is very useful
[21-Jul-2011 11:08:40] <johnf1911> that's going to add a number of machines
[21-Jul-2011 11:08:54] <johnf1911> I get database synch errors in zenmd
[21-Jul-2011 11:09:35] <johnf1911> I'm guessing I can call sync before adding each device
[21-Jul-2011 11:09:45] <johnf1911> to resync the state of my senmd shell with the database
[21-Jul-2011 11:09:48] <johnf1911> zenmd shell even
[21-Jul-2011 11:10:01] <johnf1911> so I'd call createInstance, set parameters
[21-Jul-2011 11:10:02] <johnf1911> commit
[21-Jul-2011 11:10:08] <johnf1911> call collectdevice
[21-Jul-2011 11:10:10] <johnf1911> sync
[21-Jul-2011 11:10:12] <johnf1911> and loop
[21-Jul-2011 11:10:14] <johnf1911> would that make sense?
[21-Jul-2011 11:10:19] <iancmcc> johnf1911: you're getting ConflictErrors?
[21-Jul-2011 11:10:32] <iancmcc> Highly, highly recommend using transact instead of transaction.commit()
[21-Jul-2011 11:10:53] <johnf1911> well, here's what I'm doing
[21-Jul-2011 11:11:10] <johnf1911> d = dmd.Devices.Network.Switch.Cisco.createInstance( "IP" )
[21-Jul-2011 11:11:18] <johnf1911> in a loop
[21-Jul-2011 11:11:27] <iancmcc> ok
[21-Jul-2011 11:11:28] <johnf1911> so I'm going to read a file with a bunch of machines
[21-Jul-2011 11:11:34] <johnf1911> so, with that in mind
[21-Jul-2011 11:11:38] <johnf1911> should I not be calling commit?
[21-Jul-2011 11:11:45] <johnf1911> and how can I keep my zenmd in sync?
[21-Jul-2011 11:11:48] <iancmcc> well
[21-Jul-2011 11:12:04] <iancmcc> if you're using zendmd
[21-Jul-2011 11:12:24] <iancmcc> no, either way. Here's the deal. transaction.commit() is a one-shot and can raise ConflictErrors.
[21-Jul-2011 11:12:44] <iancmcc> If your transaction lives for any length of time they'll happen on a decently busy system
[21-Jul-2011 11:12:55] <iancmcc> from ZODB.transact import transact
[21-Jul-2011 11:13:04] <iancmcc> that's a decorator.
[21-Jul-2011 11:13:23] <johnf1911> er, sorry
[21-Jul-2011 11:13:26] <johnf1911> I didn't quite follow that
[21-Jul-2011 11:13:27] <iancmcc> Wrap a function with it, and that function will be an isolated transaction that will retry up to five times in case of conflicterror
[21-Jul-2011 11:13:35] <johnf1911> ah, ok
[21-Jul-2011 11:13:49] <iancmcc> ConflictErrors happen, that's the way of things, and the best thing to do is retry
[21-Jul-2011 11:13:57] <nyeates> dpetzel: so you have one machine, many java processes, same name for each process, different ports for each, yeah?
[21-Jul-2011 11:14:17] <dpetzel> yeah process is java.exe (multiple instances of JVM running)
[21-Jul-2011 11:14:46] <johnf1911> http://pastie.org/2248950
[21-Jul-2011 11:14:54] <johnf1911> here's the code I'm going to be calling
[21-Jul-2011 11:14:58] <iancmcc> dpetzel: I'm pretty sure you'll be fine extending OSProcess.
[21-Jul-2011 11:14:59] <johnf1911> so, I should define it in a function
[21-Jul-2011 11:15:02] <johnf1911> and then call the function?
[21-Jul-2011 11:15:12] <Evolution> wait, if a message is aged, does that mean it'll drop off after X hours?
[21-Jul-2011 11:15:22] <johnf1911> I have a large list 53 hosts to add
[21-Jul-2011 11:15:24] <Evolution> so after X hours the 'info' goes back to green?
[21-Jul-2011 11:15:57] <iancmcc> Evolution: simonjj's transform is one way to go, but if you prefer a more surgical approach:
[21-Jul-2011 11:15:58] <iancmcc> http://pastebin.com/
[21-Jul-2011 11:16:01] <johnf1911> how would I create my function?
[21-Jul-2011 11:16:04] <iancmcc> bah
[21-Jul-2011 11:16:09] <johnf1911> sorry, I'm not what you'd call a python expert
[21-Jul-2011 11:16:14] <iancmcc> http://pastie.org/2248926
[21-Jul-2011 11:16:40] <iancmcc> Evolution: POST that to /zport/dmd/Devices/path/to/devices/mydevice/events_router
[21-Jul-2011 11:16:48] <iancmcc> and it will close all info-level events for that device.
[21-Jul-2011 11:16:56] <Evolution> oh sweet.
[21-Jul-2011 11:17:00] <iancmcc> johnf1911: looking
[21-Jul-2011 11:17:08] <johnf1911> thank you very much iancmcc
[21-Jul-2011 11:17:09] <Evolution> iancmcc: can that be done via xmlrpc as well?
[21-Jul-2011 11:17:44] <Evolution> iancmcc: as my script to add them is python, if I can munge it all from the same structure, that would be ideal.
[21-Jul-2011 11:17:53] <dpetzel> iancmcc: The part Im unclear aboutis after create my extended class, wouldnt Ineed to override the standard modeler plugin? Simply extending it wont make the OSProcess be that type without other changes right?
[21-Jul-2011 11:18:20] <iancmcc> Evolution: No, not yet.The xml-rpc API has always been pretty minimal, and we haven't yet updated it to mirror the JSON interface
[21-Jul-2011 11:18:20] <pmcguire> \nick ptmcg
[21-Jul-2011 11:18:38] <pmcguire> \nick ptmcg
[21-Jul-2011 11:18:54] <Evolution> the slash goes the other way :-P
[21-Jul-2011 11:19:05] <pmcguire> oof
[21-Jul-2011 11:19:09] pmcguire is now known as ptmcg
[21-Jul-2011 11:19:29] <ptmcg> (keeps me from being confused with all the other pmcguire's out there)
[21-Jul-2011 11:20:03] <iancmcc> johnf1911: Have you considered using zenbatchload?
[21-Jul-2011 11:20:32] <iancmcc> dpetzel: Yes, that's right, you would need to make a special modeler plugin that uses your OSProcess subclass.
[21-Jul-2011 11:20:33] <johnf1911> hmm, I definately could!
[21-Jul-2011 11:20:46] <johnf1911> I need to set both the title and IP though
[21-Jul-2011 11:20:51] <johnf1911> is that possible with zenbatchload?
[21-Jul-2011 11:21:07] <iancmcc> pretty sure. Checking.
[21-Jul-2011 11:21:51] <dpetzel> iancmcc: and remove the standard one from the device class correct? If I did that, I envions needing to 'determine' which type of process it is (ones I need to monkey with, vs ones I dont) and then create them as either the standard object class, or my extended class. Can Ihave multiple OSProcesses of varying class types?
[21-Jul-2011 11:21:57] <nyeates> johnf1911: you doing this on zenoss 3.1?
[21-Jul-2011 11:22:15] <johnf1911> yes, current
[21-Jul-2011 11:22:32] <johnf1911> oh, interesting
[21-Jul-2011 11:22:33] <johnf1911> zenbatchload --sample_configs
[21-Jul-2011 11:22:48] <johnf1911> it looks like I can do it by specifying the values in an appropriate form
[21-Jul-2011 11:23:07] <iancmcc> dpetzel: Yes, remove the existing process modeler from the device class. What you're saying seems exactly right, switch on whatever condition to pick which OSProcess subclass to use. I'm pretty sure all the relationship does is verify isinstance(ob, klass), which will work fine with subclasses
[21-Jul-2011 11:23:32] <iancmcc> johnf1911: Yes, you can invoke any method, so setTitle and setManageIp
[21-Jul-2011 11:24:00] <dpetzel> <iancmcc> ok cool, I'll take a crack at it and see what I break Thanks
[21-Jul-2011 11:27:29] <johnf1911> apparently I don't use the -i option any more
[21-Jul-2011 11:27:33] <johnf1911> to specify the input file
[21-Jul-2011 11:27:34] <johnf1911> any ideas?
[21-Jul-2011 11:27:58] <johnf1911> -C perhaps
[21-Jul-2011 11:29:03] <iancmcc> it's just the first arg
[21-Jul-2011 11:29:09] <iancmcc> zenbatchload myfile
[21-Jul-2011 11:29:54] <nyeates> johnf1911: if you want to see the code, check out $ZENHOME/Products/ZenModel/BatchDeviceLoader.py
[21-Jul-2011 11:29:59] <johnf1911> awesome!
[21-Jul-2011 11:30:01] <johnf1911> it's working
[21-Jul-2011 11:30:02] <nyeates> -C is a configfile, which is NOT what you want
[21-Jul-2011 11:30:07] <johnf1911> yes, clearly
[21-Jul-2011 11:30:09] <iancmcc> -C is for config files, like zenbatchload.conf, if you had one, which you wouldn't. It's inheriting that from the base class which all the daemons use.
[21-Jul-2011 11:30:13] <johnf1911> and if you use that, it comments out the entries in it
[21-Jul-2011 11:31:02] <johnf1911> ok, I've got it working perfectly
[21-Jul-2011 11:31:09] <johnf1911> thank you very much for your help iancmcc, nyeates
[21-Jul-2011 11:31:47] <nyeates> also zenbatchload --show_options is nice to see
[21-Jul-2011 11:31:57] <johnf1911> wait, I spoke too quickly
[21-Jul-2011 11:32:06] <johnf1911> for some reason it hasn't modeled the devices correctly
[21-Jul-2011 11:33:06] <nyeates> how so?
[21-Jul-2011 11:33:13] <johnf1911> it has hard specified an snmp community
[21-Jul-2011 11:33:15] <johnf1911> at the device level
[21-Jul-2011 11:33:17] <johnf1911> other than the default one
[21-Jul-2011 11:33:32] <johnf1911> I see the inheritance of the attribute
[21-Jul-2011 11:33:38] <johnf1911> and it's specific to the newly created device
[21-Jul-2011 11:33:41] <johnf1911> that's really odd
[21-Jul-2011 11:33:42] <iancmcc> It's going to check zSnmpCommunities in order to see if one matches, and if it does it'll set it.
[21-Jul-2011 11:33:58] <johnf1911> hmm, that doesn't appear to be what it has done
[21-Jul-2011 11:34:00] <iancmcc> Is the value the same as the inherited?
[21-Jul-2011 11:34:15] <johnf1911> no
[21-Jul-2011 11:34:20] <johnf1911> if it inherited it'd be fine
[21-Jul-2011 11:34:22] <nyeates> you can set the zSnmpCommunity in the zenbatcload file.....or it will use what it enherits from the device class above it
[21-Jul-2011 11:34:22] <iancmcc> ok. Well, that's good.
[21-Jul-2011 11:34:56] <johnf1911> ok, I deleted the inheritance
[21-Jul-2011 11:34:58] <johnf1911> and it fixed it
[21-Jul-2011 11:35:00] <johnf1911> but that's very odd
[21-Jul-2011 11:35:06] <johnf1911> in this case, it's not such a big deal
[21-Jul-2011 11:35:09] <johnf1911> well, actually
[21-Jul-2011 11:35:15] <johnf1911> no, I really don't like this behaviour
[21-Jul-2011 11:35:24] <johnf1911> I could hard specify it in the batch load file
[21-Jul-2011 11:35:27] <johnf1911> but then it would be local
[21-Jul-2011 11:35:28] <johnf1911> :/
[21-Jul-2011 11:35:57] <iancmcc> if you set a zproperty to the value it /would/ have had if it were inherited, it doesn't set the local value
[21-Jul-2011 11:36:06] <iancmcc> it just keeps the inheritance
[21-Jul-2011 11:36:14] <johnf1911> hmm, ok
[21-Jul-2011 11:36:22] <iancmcc> nonetheless that's odd. So you're saying it doesn't use zSnmpCommunities?
[21-Jul-2011 11:36:35] <iancmcc> it should just go down the list until one hits
[21-Jul-2011 11:36:47] <johnf1911> no, it's forced it to public
[21-Jul-2011 11:36:52] <johnf1911> which is the first in the list
[21-Jul-2011 11:37:16] <iancmcc> is public in any way valid for the device?
[21-Jul-2011 11:37:42] <johnf1911> no
[21-Jul-2011 11:38:01] <nyeates> as in, did SNMP modeled stuff show up? did memory or disk sizes show up?
[21-Jul-2011 11:38:20] <nyeates> or it failed to model anything?
[21-Jul-2011 11:38:30] <iancmcc> no, more as in, do you have public as a community string on the device at all, whether or not it gives you the access you want
[21-Jul-2011 11:38:46] <iancmcc> anyway the easy fix is just to take public out of your zSnmpCommunities
[21-Jul-2011 11:38:55] <iancmcc> put them in the order that makes sense for your environment
[21-Jul-2011 11:38:59] <nyeates> this is true
[21-Jul-2011 11:39:03] <iancmcc> up on /zport/dmd/Devices, probably
[21-Jul-2011 11:42:50] <Evolution> for remote collectors, is it possible to wrap the traffic in ssl, or tunnel it via ssh?
[21-Jul-2011 11:43:57] <simonjj> Evolution: I recommend using point-to-point OpenVPN
[21-Jul-2011 11:47:29] <Evolution> simonjj: assuming company policy won't allow that... is it possible?
[21-Jul-2011 11:48:12] <simonjj> it is not possible, but I don't really I understand why that wouldn't be allowed
[21-Jul-2011 11:48:18] <bigegor> hi all
[21-Jul-2011 11:48:18] <simonjj> all you need to is a single port
[21-Jul-2011 11:48:29] <simonjj> you can use any port
[21-Jul-2011 11:48:58] <simonjj> the whole setup takes around 15-30min, here is a quick writeup and the config I usually use: http://cl1p.net/openvpn/
[21-Jul-2011 11:49:18] <nyeates> Company policy wont allow open source vpn implementation? they trust proprietary more maybe? heh
[21-Jul-2011 11:49:47] <Evolution> I'm thinking it's because they fear laws in other countries about encryption and data protection.
[21-Jul-2011 11:49:51] <simonjj> it's like saying we don't allow SSH
[21-Jul-2011 11:49:56] <Evolution> we have locations in some uh... interesting places.
[21-Jul-2011 11:50:09] <Evolution> simonjj: you joke....
[21-Jul-2011 11:50:31] <simonjj> no, seriously, the new version of openssh can establish fill VPNs as well
[21-Jul-2011 11:50:37] <simonjj> you need nothing but SSH
[21-Jul-2011 11:51:01] <simonjj> http://www.perturb.org/display/entry/770/
[21-Jul-2011 11:51:10] <simonjj> you guys allow SSH?
[21-Jul-2011 11:51:19] <Evolution> in some locations.
[21-Jul-2011 11:51:27] <Evolution> others, local law is sketchy.
[21-Jul-2011 11:51:55] <simonjj> well, I am sorry to say that without some other means there is no way to encrypt the traffic....
[21-Jul-2011 11:52:14] <simonjj> but honestly, what's the alternative? sneaker-net ?
[21-Jul-2011 11:52:45] <simonjj> you call a guy and he types and reads while wearing a phone headset in front of the physical terminal ?
[21-Jul-2011 11:52:57] <Evolution> heh
[21-Jul-2011 11:53:10] <Evolution> large company == strange policies.
[21-Jul-2011 11:53:27] <Evolution> I don't make'em, I don't break'em... I just try to work aroudn them when I have to.
[21-Jul-2011 11:55:38] <nyeates> Thanks all for attending dev hour
[21-Jul-2011 11:55:45] <nyeates> See you all around
[21-Jul-2011 11:56:35] <simonjj> Evolution: I hear you....
[21-Jul-2011 11:56:50] <simonjj> maybe we should talk about what _is_ allowed first
[21-Jul-2011 11:58:35] <Evolution> I need to dig into that a bit more.
[21-Jul-2011 11:58:52] <Evolution> I'm still reasonably new and monitoring was something I got handed.
[21-Jul-2011 11:59:02] <Evolution> apparently it's been an issue here for quite a while.
[21-Jul-2011 11:59:17] <Evolution> (F the new guy or something)
[21-Jul-2011 12:00:01] [disconnected at Thu Jul 21 12:00:01 2011]
[21-Jul-2011 12:00:02] [connected at Thu Jul 21 12:00:02 2011]
[21-Jul-2011 12:00:21] [zenoss-logger (logger bot) has joined #zenoss]
[21-Jul-2011 12:00:24] <nyeates> maybe deeper issues if SSH is not allowed internally. Maybe see if you can get an exception from your security/network team
[21-Jul-2011 12:01:08] <Evolution> internally it's fine. it's crossing zones (country borders) where they get all sideways on stuff.
[21-Jul-2011 12:10:04] <johnf1911> iancmcc: you were entirely right, when I set it to the same value it didn't have the local value set