Timestamps on rrd files stopped updating in 3.1

Up to Discussions in zenoss-users

6204 Views 11 Replies Latest reply: Apr 29, 2011 7:51 AM by jcurry

jcurry

1,021 posts since
Apr 15, 2008

Currently Being Moderated

Apr 27, 2011 11:34 AM

This area has done the round several times before, see

thread/9438

for one.

I have Zenoss 3.1 on OpenSuse 10.3. My graphs are fine. The data inside the rrd files is updating fine. But the modified timestamps on the rrd files are not changing. I updated from 3.0.3 to 3.1 on March 11th. Many of my rrd file timestamps are March 28th or 29th - but not all. Newly created rrd files have the timestamp of creation but that timestamp should change every 5 minutes as the data gets updated (and it DOES get updated).

I am concerned that the file cleanup code in zenperfsnmp that deletes "old" rrd files, may delete my data files after 30 days. I know this issue used to appear on some Linux distros in the past and some mods were done to zenperfsnmp but not quite sure we are at with the code at present????

This Virtual Machine got forked long back from a Zenoss 2.5.2 where rrd file timestamps ARE still being updated as data is entered - the Ops Sys is identical; it's just the Zenoss that changed.

Anyone else seeing this?

Cheers,

Jane

Like (0)

Tags: rrd, file, zenperfsnmp, 3.1, timestamp

nilie
372 posts since
May 27, 2010

Currently Being Moderated

1. Apr 27, 2011 1:42 PM (in response to jcurry)
Re: Timestamps on rrd files stopped updating in 3.1

Jane,

On my Zenoss v3.1.0 the timestamps are getting updated every 5 minutes as expected. I'm running it under CentOS v5.6 64bit on a physical server.

Are you by chance using RRD caching ? Did you noticed if there are any other files affected besindes RRD files ?

Report Abuse

Like (0)
jcurry
1,021 posts since
Apr 15, 2008

Currently Being Moderated

2. Apr 27, 2011 1:45 PM (in response to nilie)
Re: Timestamps on rrd files stopped updating in 3.1

Can you help me with how to find out whether I am using RRD caching?
Cheers,
Jane

Report Abuse

Like (0)
nilie
372 posts since
May 27, 2010

Currently Being Moderated

3. Apr 27, 2011 1:57 PM (in response to jcurry)
Re: Timestamps on rrd files stopped updating in 3.1

You should look for a daemon called rrdcached running ( ps -A | grep rrdcached ). It does not seem to come with Zenoss by default so if you or someone else did not install it on purpose I don't think you have it on your system.

Report Abuse

Like (0)
jcurry
1,021 posts since
Apr 15, 2008

Currently Being Moderated

4. Apr 27, 2011 2:31 PM (in response to nilie)
Re: Timestamps on rrd files stopped updating in 3.1

Nope - no rrdcached.
Cheers,
Jane

Report Abuse

Like (0)
jhelgesen
64 posts since
Dec 10, 2007

Currently Being Moderated

5. Apr 27, 2011 4:47 PM (in response to jcurry)
Re: Timestamps on rrd files stopped updating in 3.1

Hi Jane.

If the timestamps are not updated, but the file _is_ updated, I guess I'd suspect the filesystem. If you "stat" one of the files (stat sysUpTime_sysUpTime.rrd), it ought to list out the access, change, and modify times. When I test this (2.5.2 on opensuse11.2 and 3.1.0 on opensuse11.4) all three times are the same. Do you see different times on your system?

--Jay

Report Abuse

Like (0)
jcurry
1,021 posts since
Apr 15, 2008

Currently Being Moderated

6. Apr 27, 2011 5:40 PM (in response to jhelgesen)
Re: Timestamps on rrd files stopped updating in 3.1

So I see different times:
zenoss@zen3:/usr/local/zenoss/zenoss/perf/Devices/zen3.class.example.org> date
Wed Apr 27 22:34:24 BST 2011

zenoss@zen3:/usr/local/zenoss/zenoss/perf/Devices/zen3.class.example.org> stat ssCpuIdle_ssCpuIdle.rrd
File: `ssCpuIdle_ssCpuIdle.rrd'
Size: 66208           Blocks: 144        IO Block: 4096   regular file
Device: 802h/2050d      Inode: 989546      Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1002/ zenoss)   Gid: ( 1000/ zenoss)
Access: 2011-04-27 22:33:32.000000000 +0100
Modify: 2011-04-27 15:36:26.000000000 +0100
Change: 2011-04-27 15:36:26.000000000 +0100

The modify/change time I can well believe is when I made an update to the template affecting this file. My graphs show good data upto 10:30-ish this evening, as per the access time but why hasn't the modify / change time changed????

Cheers,
Jane

Report Abuse

Like (0)
jskeane
67 posts since
Sep 15, 2010

Currently Being Moderated

7. Apr 27, 2011 7:16 PM (in response to jcurry)
Re: Timestamps on rrd files stopped updating in 3.1

Jane,

I wonder if the timestamps are "stuck" because some process is holding the files open. What does lsof think of your file? How does that change with Zenoss stopped?

Other potentially edifying experiments:

Stop Zenoss, then start it up again. Do the timestamps update to the startup time?

Stop Zenoss. Rename one of the files, appending _. Rename the file again, removing the _. This will give it a new inode. Start Zenoss, and look at the timestamps.

I hope this helps.

Regards,

jskeane

Report Abuse

Like (0)
jhelgesen
64 posts since
Dec 10, 2007

Currently Being Moderated

8. Apr 28, 2011 10:05 AM (in response to jcurry)
Re: Timestamps on rrd files stopped updating in 3.1

hmm... certainly possible that a file handle someplace didn't get closed, hanging up the 'modified' date.

What happens if the zenoss user manually modifies an RRD file?

zenoss@zenoss:....../os/filesystems/C__ Label_ Serial Number 5865c6ad> ls -la
total 44
drwxr-x--- 2 zenoss zenoss 4096 2011-04-28 09:59 .
drwxr-x--- 5 zenoss zenoss 4096 2009-12-29 15:31 ..
-rw-r--r-- 1 zenoss zenoss 35432 2011-04-28 09:57 usedBlocks_usedBlocks.rrd
zenoss@zenoss:....../os/filesystems/C__ Label_ Serial Number 5865c6ad> rrdtool lastupdate usedBlocks_usedBlocks.rrd
ds0

1303999057: 2266756.0
zenoss@zenoss:....../os/filesystems/C__ Label_ Serial Number 5865c6ad> rrdtool update usedBlocks_usedBlocks.rrd N:2266757.0
zenoss@zenoss:....../os/filesystems/C__ Label_ Serial Number 5865c6ad> rrdtool lastupdate usedBlocks_usedBlocks.rrd
ds0

1303999247: 2266757.0
zenoss@zenoss:....../os/filesystems/C__ Label_ Serial Number 5865c6ad> ls -la
total 44
drwxr-x--- 2 zenoss zenoss 4096 2011-04-28 09:59 .
drwxr-x--- 5 zenoss zenoss 4096 2009-12-29 15:31 ..
-rw-r--r-- 1 zenoss zenoss 35432 2011-04-28 10:00 usedBlocks_usedBlocks.rrd

--Jay

Report Abuse

Like (0)
jcurry
1,021 posts since
Apr 15, 2008

Currently Being Moderated

9. Apr 28, 2011 1:25 PM (in response to jskeane)
Re: Timestamps on rrd files stopped updating in 3.1

lsof thinks nothing has any of my rrd files held open

I have stopped and started both Zenoss and the entire machine without managing to get updated timestamps.

Tried your trick of renaming the file but that didn't change the inode number on my system. Tried copying it, deleting the original and copying back. No change. Obviously it gets a new time stamp when you recreate the file but new data has now been going into the file for 15 minutes but the file timestamp has not updated since creation.

This behavious is the same across ALL my rrd files - it's not just the odd one that is "stuck".

Cheers,
Jane

Report Abuse

Like (0)
jcurry
1,021 posts since
Apr 15, 2008

Currently Being Moderated

10. Apr 29, 2011 7:51 AM (in response to jhelgesen)
Re: Timestamps on rrd files stopped updating in 3.1

Neat idea. Tried it. rrdtool lastupdate sees a manual update - as it sees the zenperfsnmp updates - but the timestamp on the file doesn't budge.

Incidentally, it is not just SNMP-fed rrd files that don't update the timestamp. I have a Command template that is distributed across several systems and the resulting rrd files show just the same odd behaviour as the SNMP ones.

--------

Another update. There is a known bug in rrdtool that is tickled by some old Linux kernel distros - http://oss.oetiker.ch/rrdtool-trac/ticket/193 and http://oss.oetiker.ch/rrdtool/forum.en.html#nabble-td3898329 and http://oss.oetiker.ch/rrdtool/forum.en.html#nabble-td1618319 . There is a related append in the forum here at message/46079#46079 .

It appears that when rrdtool is built, there are options in the configure script to check whether your Op Sys has the bug and provide a workaround.I am running on an old OpenSuSE 10.3.

The other thing that has changed for me since I noted this lack of file timestamp update was an update for Zenoss 3.0.3 to 3.1. I am wondering if this update changed the build of rrdtool - I now have rrdtool 1.3.9 - or whether the Zenoss 3.1 build included a slightly different variant of configure when building rrdtool.

Anyway - the result is that the timestamps are not updated and after a month you will start to lose rrd files. They get cleaned up (deleted) by zenperfsnmp and they are then recreated when zenperfsnmp finds they don't exist. As suggested in the forum append, a simple script in cron to touch all rrd files will prevent this happening. I have the following in the zenoss user's crontab:

27 2 * * 6 find /usr/local/zenoss/zenoss/perf/Devices/ -name "*.rrd" -execdir touch '{}' +

Cheers,
Jane

Report Abuse

Like (0)
jhelgesen
64 posts since
Dec 10, 2007

Currently Being Moderated

11. Apr 29, 2011 1:12 PM (in response to jcurry)
Re: Timestamps on rrd files stopped updating in 3.1

Very good, when the manual update still failed you knew it wasn't a python problem. It seems unlikely the APIs or features of RRDtool that Zenoss uses would change on a minor point upgrade... I'd be tempted to backrev the thing and see if the problem's corrected.

My copy of Zenoss2.5.2 says it's got RRDtool 1.3.8, you should be able to pull that from your 2.5.2 copy over to the zenoss user's home directory (or tmp, someplace not in the search path), and try the manual update again with the older rrdtool. If it works, backup rrdtool139 and drop in the rrdtool138?

--Jay

Report Abuse

Like (0)

Go to original post

Legend

Correct Answers - 4 points
Helpful Answers - 2 points

Apr 27, 2011 11:34 AM

Timestamps on rrd files stopped updating in 3.1

Actions

More Like This

Incoming Links

Legend