Archived community.zenoss.org | full text search
Skip navigation
1 2 Previous Next 61369 Views 26 Replies Latest reply: Apr 6, 2010 1:38 PM by WebGuyInternet RSS
RyanK Newbie 5 posts since
Dec 16, 2009
Currently Being Moderated

Dec 16, 2009 6:23 PM

zenperfsnmp is deleting current RRD files

We have a very basic install of Zenoss that we are using for evaluation purposes. Its running 2.5.0 (on CentOS, installed via the stack installer, if that matters).
We have added several machines with the hopes that we can begin to track performance data over time. However, I went to present some of what Zenoss could potentially do for us and found that our RRD files only went back a few weeks, despite being running for several months.
After digging, I found entries in zenperfsnmp.log for each RRD file:
zenperfsnmp.log.2:2009-11-29 03:22:32,554 WARNING zen.zenperfsnmp: Deleting old RRD file: /usr/local/zenoss/zenoss/perf/Devices/testserver/os/interfaces/MS TCP Loopback interface/ifOutOctets_ifOutOctets.rrd
That coincides with when the data was wiped out. Looking at that file produces:
# stat "/usr/local/zenoss/zenoss/perf/Devices/testserver/os/interfaces/MS TCP Loopback interface/ifOutOctets_ifOutOctets.rrd"
  File: `/usr/local/zenoss/zenoss/perf/Devices/testserver/os/interfaces/MS TCP Loopback interface/ifOutOctets_ifOutOctets.rrd'
  Size: 35296           Blocks: 80         IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 6187281     Links: 1
Access: (0644/-rw-r--r--)  Uid: (  501/  zenoss)   Gid: (  501/  zenoss)
Access: 2009-12-16 17:13:31.000000000 -0600
Modify: 2009-11-29 03:22:33.000000000 -0600
Change: 2009-11-29 03:22:33.000000000 -0600
so, sure enough.. the modification date of the files are not being updated, so when the cleanup function within zenperfsnmp.py runs, it appears as if these files are not being modified. I tried searching and it appears as if nobody else is having this issue and I'm not really sure what the best way to go about dealing with this is. I would expect that the RRD tool would simply be writing these files and the filesystem (which doesn't have nomtime set) would update the modification date, but perhaps some version of RRD is keeping the files open or something? I've tested that when I edit files they indeed get their modification dates updated so, I'm kinda out of ideas at this point.
Just a wild guess... I could easily write a script that will run through and touch all the rrd files to update the modification dates, but that seems to be just a crude workaround that I'd rather avoid if possible.
Thanks for the help!
  • jmp242 ZenossMaster 4,060 posts since
    Mar 7, 2007
    Currently Being Moderated
    1. Dec 17, 2009 8:41 AM (in response to RyanK)
    Re: zenperfsnmp is deleting current RRD files

    First, I'd upgrade to the latest version of the major version you're

    running, in this case, to 2.5.1. Many many bugs have been fixed. You

    should be able to do this without losing anything new, but backups are

    always a good idea.

     

    Are these files actually being updated, that is, do you see new graph

    data? If so, I'd probably open a ticket

    http://dev.zenoss.com/trac/wiki/HowToAddTicket

    to at least report the bug/oddity, unless someone else has more info

    that I do on this.

    --

    James Pulver

    Information Technology Area Supervisor

    LEPP Computer Group

    Cornell University

     

     

     

    RyanK wrote, On 12/16/2009 6:24 PM:

    We have a very basic install of Zenoss that we are using for evaluation purposes. Its running 2.5.0 (on CentOS, installed via the stack installer, if that matters).

     

    We have added several machines with the hopes that we can begin to track performance data over time. However, I went to present some of what Zenoss could potentially do for us and found that our RRD files only went back a few weeks, despite being running for several months.

     

    After digging, I found entries in zenperfsnmp.log for each RRD file:

     

    zenperfsnmp.log.2:2009-11-29 03:22:32,554 WARNING zen.zenperfsnmp: Deleting old RRD file: /usr/local/zenoss/zenoss/perf/Devices/testserver/os/interfaces/MS TCP Loopback interface/ifOutOctets_ifOutOctets.rrd

     

    That coincides with when the data was wiped out. Looking at that file produces:

     

    1. stat "/usr/local/zenoss/zenoss/perf/Devices/testserver/os/interfaces/MS TCP Loopback interface/ifOutOctets_ifOutOctets.rrd"

      File: `/usr/local/zenoss/zenoss/perf/Devices/testserver/os/interfaces/MS TCP Loopback interface/ifOutOctets_ifOutOctets.rrd'

      Size: 35296           Blocks: 80         IO Block: 4096   regular file

    Device: fd00h/64768d    Inode: 6187281     Links: 1

    Access: (0644/-rw-rr)  Uid: (  501/  zenoss)   Gid: (  501/  zenoss)

    Access: 2009-12-16 17:13:31.000000000 -0600

    Modify: 2009-11-29 03:22:33.000000000 -0600

    Change: 2009-11-29 03:22:33.000000000 -0600

     

     

    so, sure enough.. the modification date of the files are not being updated, so when the cleanup function within zenperfsnmp.py runs, it appears as if these files are not being modified. I tried searching and it appears as if nobody else is having this issue and I'm not really sure what the best way to go about dealing with this is. I would expect that the RRD tool would simply be writing these files and the filesystem (which doesn't have nomtime set) would update the modification date, but perhaps some version of RRD is keeping the files open or something? I've tested that when I edit files they indeed get their modification dates updated so, I'm kinda out of ideas at this point.

     

    Just a wild guess... I could easily write a script that will run through and touch all the rrd files to update the modification dates, but that seems to be just a crude workaround that I'd rather avoid if possible.

     

    Thanks for the help!

     

    >

  • PaulH Rank: White Belt 7 posts since
    Feb 15, 2010
    Currently Being Moderated
    2. Feb 15, 2010 4:23 PM (in response to jmp242)
    Re: zenperfsnmp is deleting current RRD files

    Was a fix found for this problem?  I am running into the same issue. I have a basic install of Zenoss Ver. 2.5.1 running on Ubuntu server 8.10, and the RRD files are getting periodically deleted as described earlier.  I would love to know what can be done to prevent this from happening.

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009
    Currently Being Moderated
    4. Feb 23, 2010 4:59 PM (in response to RyanK)
    Re: zenperfsnmp is deleting current RRD files

    Have you guys checked in the Event Manager section to make sure that the value for "Delete Historical Events Older Than (days)" is set to a correct value?  I'm not sure whether it actually has anything to do with RRDs though, it may only affects the events stored in the MySQL database.  The RRD data defaults to being stored for a year I think.

  • PaulH Rank: White Belt 7 posts since
    Feb 15, 2010
    Currently Being Moderated
    6. Feb 23, 2010 10:05 PM (in response to RyanK)
    Re: zenperfsnmp is deleting current RRD files

    Thanks RyanK for the response. Yes I wound up adding an entry to the cron also  -  in my case it executes once a week.  My entry calls a script and is not as elegant as your one liner.  It is good to know that this seems to do the trick.  I was really dissappointed to see a whole bunch of data wiped out.  One of the biggest reasons I need these graphs is to see how things evolve over time.  30 days is not a very big window.

     

    In case anyone else comes along and looks at this, I can confirm this problem is exactly as RyanK describes it.  I dug into quite a bit hoping for a true fix rather than patching things periodically with the cron job.  I am pretty sure the following link gets to the heart of the problem which seems to be a kernel issue in some OS flavors.  That would explain why not everyone experiences this.  The root of the problem seems to be in the implementation of the mmap function in certain kernels.  RRDTool makes use of mmap.

     

    http://oss.oetiker.ch/rrdtool-trac/ticket/193

  • terrys Rank: White Belt 62 posts since
    Nov 27, 2009
    Currently Being Moderated
    7. Feb 24, 2010 10:15 AM (in response to PaulH)
    Re: zenperfsnmp is deleting current RRD files

    How about using 'lsof' to determine if a file is being held open by RRDTool and only delete those that are not open?

    Something like (untested):

     

    #!/bin/sh

    # Find rrd files with modification time older than 60 days

    for i in `find $ZENHOME/perf -mtime +60 -name *.rrd` ; do

      if lsof $i > /dev/null; then

        # file is open; could do a touch on it to update the timestamp if that helps the find command's accuracy

      else

        # file isn't open; candidate for deletion

        rm $i

    done

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009
    Currently Being Moderated
    9. Feb 26, 2010 9:41 AM (in response to RyanK)
    Re: zenperfsnmp is deleting current RRD files

    Did this just start with 2.5.x?  If so, did anyone think to maybe log a Trac ticket for this?

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009
    Currently Being Moderated
    10. Feb 26, 2010 10:53 AM (in response to RyanK)
    Re: zenperfsnmp is deleting current RRD files

    I asked Matt Ray about this and he dug up this ticket: http://dev.zenoss.org/trac/ticket/6137

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009
    Currently Being Moderated
    11. Feb 26, 2010 11:11 AM (in response to RyanK)
    Re: zenperfsnmp is deleting current RRD files

    There must be something that you guys have in common which is causing the timestamps not to be updated.

     

    What distro are each of you running?

    What filesystem are you using?

  • PaulH Rank: White Belt 7 posts since
    Feb 15, 2010
    Currently Being Moderated
    12. Feb 26, 2010 11:50 AM (in response to Ryan Matte)
    Re: zenperfsnmp is deleting current RRD files

    Yes the link which was posted earlier

     

    http://oss.oetiker.ch/rrdtool-trac/ticket/193

     

    seems to point in the direction that this problem is caused by a faulty implementation of mmap in certain distros.

    I am using Ubuntu 8.10 and the file system is ext3.

     

    I do have an XFS file system that I could move the rrd files to.  When I get a chance I will try this and see if that helps and report back.

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009
    Currently Being Moderated
    13. Feb 26, 2010 11:53 AM (in response to PaulH)
    Re: zenperfsnmp is deleting current RRD files

    I'm using Ubuntu 8.04 with ext3 and I don't have this issue.  Do you have any custom filesystem settings or anything?  Also, http://oss.oetiker.ch/rrdtool-trac/ticket/193 is patched in the version of RRDTool used in 2.5.1.

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009
    Currently Being Moderated
    14. Feb 26, 2010 12:03 PM (in response to PaulH)
    Re: zenperfsnmp is deleting current RRD files

    Also, is anyone who is having the problem messing with the ext3 atime setting.  Is everyone using ntp?

1 2 Previous Next

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points