Wiki : Monitoring the status of NFS shares with Zenoss : Comments

RE: Monitoring the status of NFS shares with Zenoss

community@zenoss.org — Tue, 22 Feb 2011 23:37:50 GMT

Thanks for the useful tip! Is the event supposed to clear when it's available again? I've tested it out by modeling the device while the NFS filesystem is mounted and then remodeling after it is unmounted and it creates an alert as expected. However, after I remount the filesystem and remodel the device, the alert remains and isn't cleared. Thanks!

RE: Monitoring the status of NFS shares with Zenoss

community@zenoss.org — Wed, 23 Feb 2011 16:51:47 GMT

The event won't auto-clear. This is because I'm manipulating a debug event. There are no actual separate clear events for debug events. The auto-clearing of them is hardcoded in to Zenoss. Since I'm changing it from a debug event to a higher severity event the debug auto-clear system no longer recognizes it as a debug event and therefore no longer clears it (if that makes sense to you).

RE: Monitoring the status of NFS shares with Zenoss

community@zenoss.org — Thu, 31 Mar 2011 07:32:10 GMT

Ive found that the OID index value for each file system can change as snmpd is restarted or file systems are mounted/un-mounted. While this changes, zenoss still using the index values found during the last device model.

This throws false positive events. Ive made some changes to make this transform a bit more active in controlling these anomalies.

Ive also found that snmpd can report the wrong state, in these cases you have to restart snmpd & re-model.

Now, Im no python hack so go easy on me.

The changes will debug to a log file, if zenoss user can write there; snmp get the live list of devices from snmpd on the device to check it reports its mounted, if so clear the previous event and remodel the device.

I dont think the re-model is working, however.

# http://community.zenoss.org/docs/DOC-4673
# raise events on unavailable NFS shares
# uses an OID.index to get FS stats, 
#  if it gets a failure assumes the FS is no longer mounted
# if the FS is remounted it gets a new index value, so the old index still fails
#  but zenoss doesnt update mapping of FS to index value until next model
# if FS OID failed, check against current mounted list from snmpget
#  if ok, FS index value has changed, ignore event
import re
import os

if re.search('1.3.6.1.2.1.25.2.3.1.6', evt.summary) and evt.severity == 1:
 p = re.compile('\"(.*)\"')
 m = p.search(evt.summary)
 share = m.group(1)
 log = open('/var/log/local/zen_NFSdown_transform.log', 'a')
 os.system('date >> /var/log/local/zen_NFSdown_transform.log')
 log.write('Summary  = ' + evt.summary + '\n')

 if share:
  d = dmd.Devices.findDevice(evt.device)

  for f in d.os.filesystems():
   if re.match(f.mount, share) and re.match('networkDisk', f.type):
    log.write('> import subprocess\n')
    import subprocess
    log.write('> get list of FSs via snmp\n')
    current_list = subprocess.Popen(['/bin/sh','-c','/usr/bin/snmpwalk -v 2c -c aspectro ' + evt.device + ' .1.3.6.1.2.1.25.2.3.1.3 | awk \'$NF ~ /^\// {print $NF}\' '], stdout=subprocess.PIPE).communicate()[0]
    #log.write('> mounted filesystems = ' + current_list + '\n')

    # check to see if its still mounted and snmpd uses a diff OID index value
    sharenotmounted = 1 # true
    for fs in current_list.split():
     log.write('>> Checking ' + share + ' with fs : ' + fs + '\n')
     if re.match(share, fs):
      log.write('>>> Share is still mounted : ' + fs + '\n')
      sharenotmounted = 0 # false

    if sharenotmounted:
     log.write('>> Not mounted, convert trap to event\n')
     evt.summary = 'NFS share %s is not unavailable.' % (share)
     evt.message = 'NFS share %s is not unavailable.' % (share)
     evt.eventClass = '/Perf/Filesystem'
     evt.component = share
     evt.severity = 5
     f.lockFromDeletion()
     d.pushConfig()
     txnCommit()

    else:
     # clear event
     events = dmd.ZenEventManager.getEventList()
     for e in events:
      if e.eventClass == '/Perf/Filesystem' and e.device == evt.device and e.component == share and e.severity == 5 and re.match('^NFS share .* is not unavailable.*', e.summary):
       log.write('>>> Clear event : ' + e.evid + '\n')
       dmd.Events.manage_deleteEvents(e.evid)

     # remodel device to correct OIDs
     log.write('>> Remodel device : ' + evt.device + '\n')
     evt.device.collectDevice()
     log.write('>> Remodel complete\n')

 log.write('\n')

RE: Monitoring the status of NFS shares with Zenoss

community@zenoss.org — Thu, 31 Mar 2011 15:00:10 GMT

You can't do evt.device.collectDevice(). You're trying to run a function against a string event property. You need to do...

d = dmd.Devices.findDevice('evt.device')

d.whatever()

RE: Monitoring the status of NFS shares with Zenoss

community@zenoss.org — Thu, 31 Mar 2011 16:04:08 GMT

Sorry, that should have been:

d = dmd.Devices.findDevice(evt.device)

not

d = dmd.Devices.findDevice('evt.device')

RE: Monitoring the status of NFS shares with Zenoss

community@zenoss.org — Mon, 04 Apr 2011 10:49:29 GMT

Thanks Ryan, that was working, but causing zenhub to pause/hang while modeling a device.

ive removed that section of code and will have to wait for the next scheduled device model.

RE: Monitoring the status of NFS shares with Zenoss

community@zenoss.org — Tue, 29 Nov 2011 18:24:28 GMT

Does this also work on the UNIX/Linux side? I ask since mounting NFS is a bit different as the mount point is a local folder and not explicitly listed as a NFS Folder.

so when viewing it under the filesystems section is simply shows as a /foldername.

PS: I use the enterprise version 3.2

RE: Monitoring the status of NFS shares with Zenoss

community@zenoss.org — Tue, 29 Nov 2011 19:30:07 GMT

This was specifically written for UNIX/Linux running net-snmp. As long as net-snmp is identifying the filesystem as type networkDisk it will work.

RE: Monitoring the status of NFS shares with Zenoss

community@zenoss.org — Tue, 29 Nov 2011 21:00:17 GMT

Thanks for the quick reply. I asked because we use /Server/SSH/ & /Server/SSH/AIX instead of the device class that uses SNMP /Servers/AIX or /Servers/Linux

I was going to try applying the /perf/snmp to the SSH Device class and see if it models.

RE: Monitoring the status of NFS shares with Zenoss

community@zenoss.org — Tue, 29 Nov 2011 21:16:57 GMT

It probably won't work unless you actually have net-snmp installed and configured on your AIX box (but I hear that's quite flaky). You'd be better off having some command datasource that actually checks whether or not the filesystems are still mounted or not.

RE: Monitoring the status of NFS shares with Zenoss

community@zenoss.org — Tue, 29 Nov 2011 21:27:05 GMT

Yeah I noticed that with AIX (thanks for the tip ) and polling SNMP. a majority of our Oracle Database servers are AIX based, with some RHEL 5.4/Oracle.

Ill keep poking around then to see if I come across a solution.