RE: Process monitoring with Zenoss

community@zenoss.org — Sun, 29 Apr 2012 15:57:24 GMT

I have been asked some excellent questions by someone who duplicated the work to use the net-snmp agent to monitor failed processes and to send traps when a process was "unhealthy" (basically pages 7-11 of the paper). On top of that, he had used the Zenoss trap configuration and event transforms to restart a failed process (pages 29-35). He found that the event transform on page 31 wasn't working:

"I kept getting the following error in zenhub.log:"

2012-04-20 14:13:42,525 WARNING zen.Events: Error processing

transform/mapping on Event Class

/App/Stop/instances/1.3.6.1.4.1.1234.123

Problem on line 6: AttributeError: Event instance has no attribute 'errorFlag'

if evt.errorFlag==1:

Transform:

0 for attr in dir(evt):

1 if attr.startswith('1.3.6.1.4.1.2021.2.1.100'):

2 evt.index=attr.replace('1.3.6.1.4.1.2021.2.1.100.','')

3 evt.process_name=getattr(evt,'1.3.6.1.4.1.2021.2.1.2.'+evt.index)

4 evt.errorFlag=getattr(evt,'1.3.6.1.4.1.2021.2.1.100.'+evt.index)

5 evt.errFixCmd=getattr(evt,'1.3.6.1.4.1.2021.2.1.103.'+evt.index)

6 if evt.errorFlag==1:

7 evt.summary=evt.process_name + ' process is unhealthy'

8 evt.severity=5

9 if evt.errorFlag==0:

10 evt.summary=evt.process_name + ' process is healthy'

11 evt.severity=0

Well, for starters, I guess I have learned a bit about Python in the last two years so each of those getattr statements in lines 4,5 and 6 should have defaults. In particular on line 4, the error flag is defined by the net-snmp agent to be 0 if the process is not in error and 1 if it is in error; so a default on the getattr of 2 would not affect the logic of the net-snmp agent but would mean that the test of evt.errorFlag on lines 6 and 9, won't fail if, for some reason there was no attribute 1.3.6.1.4.1.2021.2.1.100. So lines 3-5 would be better as:

3 evt.process_name=getattr(evt,'1.3.6.1.4.1.2021.2.1.2.'+evt.index, 'No process name')

4 evt.errorFlag=getattr(evt,'1.3.6.1.4.1.2021.2.1.100.'+evt.index,2)

5 evt.errFixCmd=getattr(evt,'1.3.6.1.4.1.2021.2.1.103.'+evt.index, 'No fix cmd')

Still doesn't explain why the attributes were not matching as he assures me that the attributes were actually present. The only thing I can suggest here are typos - especially watching for the end dot on the getattr statements. Alternatively, if he had loaded up the SMI or UCD MIBs then this could have resulted in the non-match - see the rest of the discussion.

However, more interestingly, he goes on to say:

"I checked the event details, and the attributes are definitely there.

I even imported the UCD mib and tried both the transform version

above, and the same one but with named OIDs (eg: prErrorFlag, prNames

etc) as per the UCD mib, but I kept getting the same error.

In the end, I wrote the following transform myself which does seem to

work (it includes the extra code to use snmpset to restart the

'unhealthy' process):

import os

index = getattr(evt,'prIndex','0')

process_name = getattr(evt,'prNames')

errorFlag = getattr(evt,'prErrorFlag')

errFixCmd = getattr(evt,'prErrFixCmd')

if errorFlag==1:

shellcmd = '/usr/bin/snmpset -v 3 -u zenoss -l authPriv -a MD5 -A

zenoss12 -x DES -X zenoss12 ' + dev.manageIp + '

1.3.6.1.4.1.2021.2.1.102.' + str(index) + ' i 1'

os.system(shellcmd)

evt.summary = process_name + ' process is unhealthy'

evt.severity = 5

if errorFlag==0:

evt.summary = process_name + ' process is healthy'

evt.severity = 0

Now the comment about importing the UCD mib is interesting. I did not discuss this much in the paper. Personally I dislike the way Zenoss uses imported MIBs to "decode" traps (I also hate the fact that there isn't a MIB Browser built in to the product to help you setup OIDs to poll for in performance templates, but that is a different issue). I think the OID translation stuff in TRAPs is dangerous, especially if you then use transforms to process the trap varbinds.

My original transform uses OIDs to check for particular trap varbinds. I had no MIBs imported into Zenoss. If someone imports the SNMPv2-SMI mib, suddenly all my transform will fail because the code that interprets traps will try to convert oids to names - even partially - so my trap varbinds will suddenly become things like:

enterprises.2021.2.1.100 instead of 1.3.6.1.4.1.2021.2.1.100

The leading 1.3.6.1.4.1 gets translated to enterprises because the SMI (Structure of Management Information) MIB will provide that translation.

When my friend above imported the UCD MIB, the varbinds get translated right down to the trap description name so:

1.3.6.1.4.1.2021.2.1.100.1 becomes simply prErrorFlag.1

Again, if I had written my transform to match enterprises.2021.2.1.100 then it would break again as the varbind name has changed again.

My friend has then obviously made an excellent observation and asks a very pertinent question:

I noticed that the event details showed both an indexed and

non-indexed attribute for each attribute which were always the

same.....eg: there was a prErrorFlag and a prErrorFlag.2 both of which

were set to 1 when the process being monitored died. It was using the

non-indexed value that led me to the above script. Is there any reason

I shouldn't be doing it this way? Is there a reason you explicitly use

the indexed values?

He's right! The TRAP itself provides an indexed set of values where the index represents the row of the process table. So if your snmpd.conf has 3 processes then the first will be index 1, the second index 2 and so on. Why, with the UCD MIB imported, do we get TWO copies of all the varbinds, one indexed and one non-indexed and which should you use?

You need to look in $ZENHOME/Products/ZenEvent/zentrap.py for the answer. The function oid2name is what actually does the conversion of oids to textual names (note that this website may have messed up the Python indentation):

def oid2name(self, oid, exactMatch=True, strip=False):

"""

Returns a MIB name based on an OID and special handling flags.

@param oid: SNMP Object IDentifier

@type oid: string

@param exactMatch: find the full OID or don't match

@type exactMatch: boolean

@param strip: show what matched, or matched + numeric OID remainder

@type strip: boolean

@return: Twisted deferred object

@rtype: Twisted deferred object

"""

if type(oid) == type(()):

oid = '.'.join(map(str, oid))

oid = oid.strip('.')

if exactMatch:

if oid in self.oidMap:

return self.oidMap[oid]

else:

return oid

oidlist = oid.split('.')

for i in range(len(oidlist), 0, -1):

name = self.oidMap.get('.'.join(oidlist[:i]), None)

if name is None:

continue

oid_trail = oidlist[i:]

if len(oid_trail) > 0 and not strip:

return "%s.%s" % (name, '.'.join(oid_trail))

else:

return name

return oid

Note the test almost at the end of "if len(oid_trail) > 0 and not strip" - strip is a boolean parameter passed to this function.

Now look further down zentrap.py to the asyncHandleTrap function which is what actually constructs the trap.

# Decode all variable bindings. Allow partial matches and strip

# off any index values.

for vb_oid, vb_value in variables:

vb_value = self._convert_value(vb_value)

vb_oid = '.'.join(map(str, vb_oid))

# Add a detail for the variable binding.

r = self.oid2name(vb_oid, exactMatch=False, strip=False)

result[r] = vb_value

# Add a detail for the index-stripped variable binding.

r = self.oid2name(vb_oid, exactMatch=False, strip=True)

result[r] = vb_value

This is around line 485 in my Core 3.1 version. oid2name is called first with strip=False and then with a strip=True. So two event attributes are created out of 1 varbind - one with the instance included and one with the instance stripped off.

If there was always just one instance to process then we could forget about all this clever code to work out the instance. With these particular traps from the process table, you probably are safe using either the simple attribute or the indexed one.

However, go and look at the transform supplied as standard for /Net/Link/snmp_linkDown. It has some code that does something vaguely similar to the transforms discussed here:

if_index_str = getattr(evt.details, "ifIndex", None)

if if_index_str is not None and device is not None:

if_index = int(if_index_str)

for interface in device.os.interfaces():

if interface.ifindex == if_index:

evt.component = interface.id

break

And look at the screenshot of a real captured event:

This is a trap to do with interfaces and there are multiple instances of interfaces - ifIndex.1 and ifIndex.65539. Again the simple attribute ifIndex also exists and, I assume, has taken the last value. I think this demonstrates that, depending on the trap and the varbinds it sends, you cannot always rely on using the "uninstanced" attribute.

The last question asked is about "good news" traps from the net-snmp agent.

I have my /etc/snmpd/snmpd.conf file on the monitored device

configured exactly as you have done on pages 9,10 of your paper. You

mention on page 9 and 10 about not using the following line in the

monitored device's /etc/snmpd/snmpd.conf file:

monitor -u _internal -r 20 -S -e procFix "Process table event" prErrorFlag !=0

....because zenoss doesn't get to see the 'good news' SNMP trap. Well,

I don't have this line in my snmpd.conf, but I am not getting the good

news event. My version of the transform definitely works....the bad

news event is received by zenoss, the snmpset command is run back to

the monitored device, and the process which died is restarted.....at

this point I would expect a good news event to be sent to zenoss, but

it never happens....the snmpd on the monitored device never generates

the good news trap. Before I had a working transform, if I manually

did a process restart, it did send the good news trap. So it seems

that the snmpset command is somehow causing snmpd to not generate the

good news trap."

I have to agree here. I also do not get the good news trap when the transform automation has done the snmpset. The prErrFixCmd DOES run; the problem IS resolved. But in the next polling interval specified in your snmpd.conf I also do not get a "good news" trap issued, whereas if I restart a failed process manually then I DO see a "good news" trap. Fortunately, it doesn't affect the ability to raise subsequent "bad news" traps but I too would want to see the good news confirmation. Incidentally, I had the polling interval on my monitor statement originally set to 10 (seconds) and I tried upping that to 60 - but still no good news! By turning the debugging up to "Debug" on the zentrap daemon, one can see that the "good news" trap never arrives, so the problem must be with the agent. Be interesting to see if other people with different versions of the agen get different results - these latest tests are with net-snmp-5.4.1-19.6 on an Open SuSE platform.

Cheers,

Jane

Support and Services Partners : Process monitoring with Zenoss : Comments

RE: Process monitoring with Zenoss