Set dependency using wildcards in a transform

Up to Discussions in zenoss-users

12350 Views 13 Replies Latest reply: Mar 10, 2010 11:16 PM by Matthew Kitchin (public)

Matthew Kitchin (public)

170 posts since
Nov 12, 2009

Currently Being Moderated

Feb 26, 2010 5:07 PM

Sorry for the people who are basically seeing this twice or more. I've discussed it in several threads, but I'm trying to create one that is actually devoted to it. Here goes.

My wan is Verizon MPLS circuits, and apparently not compatible with Zenoss network topology modeling.
thread/12061?tstart=0

I have seen the transforms that have an entry specific to a given subnet/location, but I have 120 locations, so I would prefer not to have a line for every

location in the transform. When a router goes down, I don't want to know about a server or a WAP going down at the same location. I have not

filled out the location value on my routers, but would certainly be glad to if that could help somehow. I had tried to do it below based on our naming convention.

All my devices are named like this

TX252-RTR

TX252SRV

TX252WAP1

I had tried the code below (I didn't write it; I don't know python) , but it doesn't seem to be working.

if device.id.endswith('WAP') or device.id.endswith('SRV'):

rtr = device.id[0:5]+'-rtr-ser'

rtr_server = device.findDevice(rtr)

if rtr_server.getPingStatus() > 0:

evt.eventState = 2 # suppressed

We are getting swamped with alerts about devices behind the router being down when a circuit goes down. If there is any way I could just say, if TX252-RTR is down, ignore all TX252*** devices, this app would be absolutely wonderful for us.

Like (0)

phonegi
446 posts since
Apr 15, 2009

Currently Being Moderated

1. Feb 27, 2010 10:31 PM (in response to Matthew Kitchin (public))
Re: Set dependency using wildcards in a transform

The code is close, let's see if we can clean it up:

router = device.findDevice(device.id[0:5] + '-RTR') if not router is None and not router is device: if rtr.getPingStatus() > 0: evt._action = 'drop'

Description:
Line 1 - This takes the first five letters of the name of the device that generated the event, attaches '-RTR' and then tries to retreive the object. So if device TX252SRV generates an event, line 1 will attempt to retrieve the object named 'TX252-RTR' and assign it to the variable name router.

Line 2 - Verifys that the device object was found and that it is not a router.

Line 3 - Checks to see if the device 'router' has any 'ping down' events

Line 4 - If 'router' has ping down events, then drop this event.

Now, this still may not work. Why? Ping sequencing. There is no way to force the ping command to ping in a specific order. Consider this example with three devices - a router: R, a server: S1, another server: S2. Assume S1 and S2 are behind router R.

If the ping sequence is R, S1, S2, then the script should work. The ping to R will time out and generate an event. Next the ping to S1 will time out. However, when line 3 is executed, getPingStatus will return a value greater than 0 because R has an active 'ping down' event. The S1 'ping down' event will be dropped. The same will occur for S2.

Now, if the ping sequence is S1, S2, R, we're in trouble. The ping to S1 will generate an event; however, when line 3 is executed getPingStatus will return 0 because there is no active 'ping down' event for R yet, so the event will be recorded and you will be alerted. Same for S2. Finally R will generate a 'ping down' event.

Without control of the ping sequence, this won't work.

Report Abuse

Like (0)
Matthew Kitchin (public)
170 posts since
Nov 12, 2009

Currently Being Moderated

2. Feb 27, 2010 10:46 PM (in response to phonegi)
Re: Set dependency using wildcards in a transform

Thank you very much. I will play with this on Monday!

Report Abuse

Like (0)
phonegi
446 posts since
Apr 15, 2009

Currently Being Moderated

3. Feb 27, 2010 11:01 PM (in response to phonegi)
Re: Set dependency using wildcards in a transform

You may also want to check out this article:

blogs/zenossblog/2008/08/26/tip-of-the-month-layer-3-dependency-checker

Zenoss is supposed to perform Layer 3 dependencies, which sounds like what you're trying to do. Also search for "layer 3 depencencies" and you will get more info.

Report Abuse

Like (0)
Matthew Kitchin (public)
170 posts since
Nov 12, 2009

Currently Being Moderated

4. Feb 27, 2010 11:09 PM (in response to phonegi)
Re: Set dependency using wildcards in a transform

Thanks. I was under the impression the network topology part wasn't going to work properly because on an MPLS network, it can't model every device since some are owned by the telco. I'm on a bberry right now, but I will check this out later.

Report Abuse

Like (0)
terrys
62 posts since
Nov 27, 2009

Currently Being Moderated

5. Feb 28, 2010 8:47 AM (in response to Matthew Kitchin (public))
Re: Set dependency using wildcards in a transform

It depends on how the MPLS network operates. More than likely, the carrier is hiding its internal routers and a traceroute that you run will only show one hop across the carrier's network. In that case, the MPLS network looks like a single hop and should be indistinguishable from a single physical link.

Report Abuse

Like (0)
Matthew Kitchin (public)
170 posts since
Nov 12, 2009

Currently Being Moderated

6. Mar 1, 2010 2:58 PM (in response to phonegi)
Re: Set dependency using wildcards in a transform

I definitely want to try this traceroute utility out. Maybe this will help point to the root of my problems. I downloaded it and ran it, and got this first:
Invalid source adress.
Add your zenoss server name into /opt/zenoss/etc/zenping.conf
eg:
name yourzenossserver

No problem. I added the hostname in that format.

I get the output below whenever I try to trace to a router over a wan connection.

[zenoss@nshplmon1 bin]$ ./tracepath.py ma178-rtr-ser
Getting path from nshplmon1 to ma178-rtr-ser...
Traceback (most recent call last):
File "./tracepath.py", line 38, in ?
path = source.os.traceRoute(destination, [])
File "/opt/zenoss/Products/ZenModel/OperatingSystem.py", line 124, in traceRoute
raise TraceRouteGap("unable to trace to %s, gap at %s" % (target.id,
Products.ZenModel.Exceptions.TraceRouteGap: unable to trace to MA178-rtr-ser, gap at NSHPLMON1
[zenoss@nshplmon1 bin]$

It can ping that router
[zenoss@nshplmon1 bin]$ ping MA178-rtr-ser
PING MA178-rtr-ser.hardware (152.164.247.190) 56(84) bytes of data.
64 bytes from 152.164.247.190: icmp_seq=1 ttl=249 time=36.0 ms
64 bytes from 152.164.247.190: icmp_seq=2 ttl=249 time=35.9 ms

traceroute shows:
traceroute to MA178-rtr-ser (152.164.247.190), 30 hops max, 40 byte packets
1 union_core_sw-vlan1.hardware (10.85.0.1) 1.056 ms 1.069 ms 1.174 ms == router on same subnet as zenoss
2 dsi_rtr1.hardware (10.84.0.6) 1.166 ms 1.213 ms 1.257 ms == core router
3 (152.164.107.177) 1.454 ms 1.449 ms 1.433 ms == Verizon MPLS first hop
4 (152.164.247.189) 27.873 ms 27.892 ms 27.884 ms Verizon MPLS last hop
5 (152.164.247.190) 37.079 ms * *
Any ideas on what I may be doing wrong?

Report Abuse

Like (0)
phonegi
446 posts since
Apr 15, 2009

Currently Being Moderated

7. Mar 1, 2010 3:53 PM (in response to Matthew Kitchin (public))
Re: Set dependency using wildcards in a transform

Check out fdeckerts comments on this thread. It is complicated, but he figured out a way to create dummy routes through the "cloud" of a service providers MPLS network.

Report Abuse

Like (0)
Matthew Kitchin (public)
170 posts since
Nov 12, 2009

Currently Being Moderated

8. Mar 1, 2010 4:03 PM (in response to phonegi)
Re: Set dependency using wildcards in a transform

Thanks. I will get with my network engineer and see if thi will work for us. Should the tracepath.py have made it farther than it did though? It seems like it didn't even make it out of the local server.

Report Abuse

Like (0)
phonegi
446 posts since
Apr 15, 2009

Currently Being Moderated

9. Mar 1, 2010 4:33 PM (in response to Matthew Kitchin (public))
Re: Set dependency using wildcards in a transform

Hmmm. can you tracepath.py to your local gateway? If not, did you set the name variable to your FQDN?

Report Abuse

Like (0)
Matthew Kitchin (public)
170 posts since
Nov 12, 2009

Currently Being Moderated

10. Mar 1, 2010 5:17 PM (in response to phonegi)
Re: Set dependency using wildcards in a transform

I can now. I seem to have forgotten to let it model a rather important router.

Now it dies when leaving a core router and entering the Verizon MPLS cloud. Now maybe I can get somehwere with the trick with the fake routes so it will skip the Verizon hops. Thanks!

[zenoss@nshplmon1 bin]$ ./tracepath.py MA178-rtr-ser
Getting path from nshplmon1 to MA178-rtr-ser...
Traceback (most recent call last):
File "./tracepath.py", line 38, in ?
    path = source.os.traceRoute(destination, [])
File "/opt/zenoss/Products/ZenModel/OperatingSystem.py", line 123, in traceRoute
    return nextdev.traceRoute(target, ippath)
File "/opt/zenoss/Products/ZenModel/Device.py", line 584, in traceRoute
    return self.os.traceRoute(target, ippath)
File "/opt/zenoss/Products/ZenModel/OperatingSystem.py", line 123, in traceRoute
    return nextdev.traceRoute(target, ippath)
File "/opt/zenoss/Products/ZenModel/Device.py", line 584, in traceRoute
    return self.os.traceRoute(target, ippath)
File "/opt/zenoss/Products/ZenModel/OperatingSystem.py", line 124, in traceRoute
    raise TraceRouteGap("unable to trace to %s, gap at %s" % (target.id,
Products.ZenModel.Exceptions.TraceRouteGap: unable to trace to MA178-rtr-ser, gap at DSI_rtr1

Report Abuse

Like (0)
Matthew Kitchin (public)
170 posts since
Nov 12, 2009

Currently Being Moderated

11. Mar 5, 2010 1:23 PM (in response to phonegi)
Re: Set dependency using wildcards in a transform

I'm trying to figre out how to crate the 'dummy' routes between sites, but I'm not sure I understand.
The thread you refrerred me to says:
'on the router collectors we removed the RouteMap collector to avoid seeing the telco'
I can't find that option anywhere. If I did find it and removed it, does that mean that all dynamic route discovery would now be disabled? I hope not.
I guess I just need to tell it to get from poitn A to D and ignore B and C.
Thanks,
Matthew

Report Abuse

Like (0)
phonegi
446 posts since
Apr 15, 2009

Currently Being Moderated

12. Mar 5, 2010 1:46 PM (in response to Matthew Kitchin (public))
Re: Set dependency using wildcards in a transform

Yes, it would mean dynamic route discovery for that device would stop.

To remove the zenoss.snmp.RouteMap collector: From a device, select the main drop down > More > Collector Plugins

From the list, click the "x" next to zenoss.snmp.RouteMap

Report Abuse

Like (0)
Matthew Kitchin (public)
170 posts since
Nov 12, 2009

Currently Being Moderated

13. Mar 10, 2010 11:16 PM (in response to phonegi)
Re: Set dependency using wildcards in a transform

I think I'm just going to hold out hope that the dependency checking is improved. I don't think I want to turn off the dynamic route discovery for all my routers. I really would be ok if Zenoss just had a way to truly manually set dependencies. That is what we did in Whatsup.

Report Abuse

Like (0)

Go to original post

Legend

Correct Answers - 4 points
Helpful Answers - 2 points

Feb 26, 2010 5:07 PM

Set dependency using wildcards in a transform

Actions

More Like This

Legend