Archived community.zenoss.org | full text search
Skip navigation
84951 Views 10 Replies Latest reply: Dec 5, 2009 5:20 PM by JesseWagner RSS
HummerBoy Rank: Green Belt 116 posts since
Dec 17, 2008
Currently Being Moderated

Mar 17, 2009 8:57 PM

Mount free space is negative

I am getting random servers with negative free space on their mounts. The one I have at the moment is a SUSE linux server. I have one mount that is essentially empty but is showing 486Gb used (-4.6Gb free). The other mount points look OK. Any ideas how to fix this?
  • mwcotton Rank: Brown Belt 563 posts since
    Apr 23, 2008
    Currently Being Moderated
    1. Mar 18, 2009 9:18 PM (in response to HummerBoy)
    RE: Mount free space is negative
    My theory:
    the total size of the disk has probably changed since you last modeled the device. This value is stored in the database and it not updated at every snmp poll , consequently if the bytes used ( which is updated at each poll ) is greater than the stored amount you will get a negative value.

    If this is the way it works, I think Zenoss might need to rethink this design, with the advent on SAN, LVM, growing and shrinking file systems iit happens more and more.
  • mwcotton Rank: Brown Belt 563 posts since
    Apr 23, 2008
    Currently Being Moderated
    3. Mar 19, 2009 4:48 PM (in response to HummerBoy)
    RE: Mount free space is negative
    if you model it and wait for some poll cycles, does it still stay negative?
  • mwcotton Rank: Brown Belt 563 posts since
    Apr 23, 2008
    Currently Being Moderated
    5. Mar 20, 2009 5:26 PM (in response to HummerBoy)
    RE: Mount free space is negative
    hmmm, If I were you I would use the snmpget command with the oids that zenoss is using, ( look at the template ) and check to see if the numbers zenoss is getting is sane.
  • JesseWagner Rank: White Belt 9 posts since
    Sep 30, 2008
    Currently Being Moderated
    6. Nov 4, 2009 11:43 AM (in response to HummerBoy)
    Re: Mount free space is negative

    I have the same issue. I think the oids are getting reordered somehow as this seems to be triggered by deleting a volume off  my netapp or windows file cluster; I have the issue on both. This just started after the upgrade to 2.4.5.

     

    UPDATE: I just upgraded to 2.5 and no luck. Same issue.

  • guyverix ZenossMaster 846 posts since
    Jul 10, 2007
    Currently Being Moderated
    7. Nov 5, 2009 3:24 AM (in response to JesseWagner)
    Re: Mount free space is negative

    We get this issue from time to time in Ver 2.4.1.  Generally I am seeing this on our disk volumes that are greater than 500GB, but not always.  When it happens, I disalbe monitoring for the file system, lock it from updates, remodel, and then unlock it and model again.  I dont know why, but that usually fixes the problem for me.

     

    If it happens again soon, I will login to the device and see if Net-SNMP has gone insane in a new and unusual way.

  • guyverix ZenossMaster 846 posts since
    Jul 10, 2007
    Currently Being Moderated
    8. Nov 5, 2009 3:25 AM (in response to JesseWagner)
    Re: Mount free space is negative

    JesseWagner wrote:

     

    I have the same issue. I think the oids are getting reordered somehow as this seems to be triggered by deleting a volume off  my netapp or windows file cluster; I have the issue on both. This just started after the upgrade to 2.4.5.

     

    UPDATE: I just upgraded to 2.5 and no luck. Same issue.

    Are you restarting the SNMP daemon after deleting the volumes?

  • JesseWagner Rank: White Belt 9 posts since
    Sep 30, 2008
    Currently Being Moderated
    9. Nov 5, 2009 11:54 AM (in response to guyverix)
    Re: Mount free space is negative

    I've never had it happen on a Linux system. Only on a Windows 2008 cluster with STK luns and a Netapp.  It never happened once before 2.4.5 in the entire time I've used zenoss which is over a year.

     

    It's almost like the oids used for the volume size are getting mixed up after a volume delete; zenoss is losing track of which volumes are which and applying the wrong volume size to the wrong used blocks value. Infact I checked once after deleting a volume and all of the oids were off by one in the OS view vs doing a snmpwalk. Something to note is that we have not changed the OnTAP version on the netapp for quite a while.

  • JesseWagner Rank: White Belt 9 posts since
    Sep 30, 2008
    Currently Being Moderated
    10. Dec 5, 2009 5:20 PM (in response to JesseWagner)
    Re: Mount free space is negative

    Just to provide an update as to where I am with this I shall post what I did to resolve my issues.

     

    I replaced the netapp snmp and windows FileSystem templates; Augmented is more accurate. For windows I used WMI to get disk space and for Netapp I used ssh. I can walk anyone who cares through creating their own versions of these but to share the plugins themselves would probably requires an OK from my company's legal dept.

     

    This is just one more step in removing snmp totally from my monitoring. Snmp or atleast how zenoss implements it is very fragile thing. Long ago I repaced the process monitoring with templates. To this end I would suggest that zenoss abandon the fancy built in monitoring, or replace all of it with templates so it can be more easily replaced/removed/fixed by end users. It's hard to test this stuff properly unless you are actually running the monitoring yourself and who has every version of snmp lib/netapp/sun/EMC/windows/switch version/brand around anyways? So to this end I would suggest a more modular system removing any built in monitoring/calls and adding hooks for templates for things like Status,more OS page parts like services or processes, hardware/software. The impressive parts of zenoss for me has been the great interface, object oriented device tree, and the multi graph reporting. Most of the built in monitoring bits have not worked so well.

     

    Jesse

More Like This

  • Retrieving data ...