Archived community.zenoss.org | full text search
Skip navigation
59008 Views 5 Replies Latest reply: Dec 7, 2009 11:06 AM by jenkinskj RSS
jenkinskj Rank: Green Belt 330 posts since
Jul 30, 2009
Currently Being Moderated

Dec 1, 2009 6:03 PM

Zenoss Data Retention

Team,

 

I am looking for some advice regarding how long we can retain data for within Zenoss. Event History, Event Correlation, Trending, Availability and Performance reporting is extremely important to us.

 

We would like to retain up to 1 year's worth of data but I am not sure how this would impact Zenoss performance overall.

 

From a device perspective we would be looking to monitor about 300 network devices, 1000 servers (UNIX / Windows), 150 URL's across 5 remote data collectors. Our primary Zenoss server would contain the bulk of devices (32GB memory, 3-500GB disk configured for RAID-5).The data collectors would have 250GB disk, with 2GB of memory initially and adding more if required)

 

I am looking for configuration and backup recommendations for MySQL, ZenRRD and Zope to ensure that the data retained is kept in sync.

 

I specifically am interested in configuration recommedations related to ZenRRD files. Where is ZenRRD configured in Zenoss? I realize that this is a round robin database and wonder how easy it will be to retain 1 year worth of data within or archive data in 1 month increments (that is if we can report on it once archived).

 

If 1 year of data might put a performance strain on performance, 6 months would be considered with an option to archive the orginal 6 months worth of data to a parallel server for reporting purposes.Has anyone run into this scenario before?

 

Please let me know what if any best practices or recommendations there are out there.

 

Thank you,

Ken

  • Matt Ray Rank: Zen Master 2,484 posts since
    Apr 5, 2008
    Currently Being Moderated
    1. Dec 1, 2009 11:57 PM (in response to jenkinskj)
    Re: Zenoss Data Retention

    The default RRD settings are set under Collector->localhost->Edit->Default RRD Create Command, you can tweak that to fit your requirements.  For details about the rrdcreate command, go to http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html  The RRD files are fixed size, but writing that many data points can be quite I/O intensive.  Once you've got your servers all added and your data points in place, your file sizes would be static.

     

    You can manage your event retention under Event Manager->Delete Historical Events Older Than (days), or you can manage it with some other OLAP/archival tool to pull them straight from MySQL.

     

    There have been a few forum threads on performance and data collection, perhaps take a look at New 36G, 16CPU server or How to tweak zenoss.

     

    Good luck,

    Matt Ray

    Zenoss Community Manager

  • Falk Rank: White Belt 89 posts since
    Jul 27, 2007
    Currently Being Moderated
    4. Dec 6, 2009 5:24 AM (in response to jenkinskj)
    Re: Zenoss Data Retention

    jenkinskj wrote:

     

    We will be using 32GB RAM, 4 CPU, 3-500GB drives in a Raid 5 configuration which equates to about 600GB of data we can store. With 1500 monitors and a data retention rate of 1 year. It would be great to calculate approximately how large the database could get and how much storage is needed for RRD files. I suspect that there is no formula we can use.My management requires that we retain 1 year of data. Are there any performance issues we need to consider if we retain up to 1 year of history with the configuration mentioned?

     

     

    - Ken

     

    Hi Ken.

     

    Without being an expert in the Zenoss field in any way, I think that a Raid5 conf would bottleneck your installation.
    Those RRD files can be realy heavy on I/O, and with the overhead of a R5 things can get really ugly.

     

    I would atleast get one more hd and do RAID10.
    Well, perhaps not so much of an answer to your post, just my 2 cents

     

    --

    Regards Falk

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points