Archived community.zenoss.org | full text search
Skip navigation
Up to Discussions in zenoss-users
57391 Views 9 Replies Latest reply: Feb 19, 2010 10:22 AM by Ryan Matte RSS
mcking Newbie
mcking
5 posts since
Feb 16, 2010
Currently Being Moderated

Feb 18, 2010 11:37 AM

Questions on the Zenoss Appliance

I'm trying to do 2 things with the Zenoss Appliance.

 

  1. Back up the Zenoss Appliance using Netbackup.  The Netbackup processes run under inetd, but there is no inetd installed in the appliance or available in the conary repository.  The appliance repository also has no gcc or other dev tools available, so compiling is not an option either.  Is it OK to add another repository to conary to pull in something like xinetd?
  2. Setup a firewall.  Is there documentation available on the proper way to setup iptables in rPath appliances?
  • Andrea Consadori ZenossMaster
    Andrea Consadori
    863 posts since
    Feb 11, 2008
    Currently Being Moderated
    1. Feb 18, 2010 1:45 PM (in response to mcking)
    Re: Questions on the Zenoss Appliance

    rpath appliance is good for test not for production..

    instead install a centos vm and zenoss rpm on it... if you read the install guide is very easy and you will have a powerfoul installation.

  • mcking Newbie
    mcking
    5 posts since
    Feb 16, 2010
    Currently Being Moderated
    2. Feb 18, 2010 1:59 PM (in response to Andrea Consadori)
    Re: Questions on the Zenoss Appliance

    The problem with that is that we already tried that route and performance was abysmal, even after a week of tuning the CentOS VM and Zenoss.  The appliance worked immediately and let us import all 650 devices without any performance issues.  The powers that be want the appliance since they need monitoring yesterday if not sooner and can't wait for us to troubleshoot why the CentOS VM is vastly outperformed by the Appliance.

  • Ryan Matte ZenossMaster
    Ryan Matte
    653 posts since
    Mar 26, 2009
    Currently Being Moderated
    3. Feb 18, 2010 3:29 PM (in response to mcking)
    Re: Questions on the Zenoss Appliance

    As Andrea stated, the appliance is fine for demo/testing but absolute garbage for production.  I am monitoring 352 devices (17.5k datapoints) on my Zenoss install and it's barely even breathing hard.  Here are the options that you need to change to enhance performance: docs/DOC-2521.

     

    Install whatever distro you want (CentOS and Ubuntu Server are the best tested distros), then install Zenoss, and perform those tweaks.

  • mcking Newbie
    mcking
    5 posts since
    Feb 16, 2010
    Currently Being Moderated
    4. Feb 18, 2010 5:06 PM (in response to Ryan Matte)
    Re: Questions on the Zenoss Appliance

    If this is the case, then why is the Appliance (with NO tweaks to any of the Zenoss config files) vastly outperforming a CentOS-based VM on the same hardware?  In both cases, the VM ran with 1vCPU (we did test the CentOS VM with up to 4 vCPUS) and 3GB RAM.  I spent a week tweaking disk and network I/O on the CenOS install and still had horrible performance.

     

    With the CentOS/RPM install, we couldn't get more than a dozen Cisco and Juniper devices into Zenoss without bringing the VM to its knees.  The console was unresponsive for up to 30 seconds at a time and the Zenoss web GUI would time out if there was a batch load of devices.  With the Appliance, we have 650 devices with 208000 datapoints, and there is never contention for resources and the CLI and web GUI are very responsive, even when the waitio on the system is upwards of 80%!

     

    The CentOS VM was a bog-standard 32-bit install from the network-install ISO with no additional packages above the base OS (JEOS).

  • Ryan Matte ZenossMaster
    Ryan Matte
    653 posts since
    Mar 26, 2009
    Currently Being Moderated
    5. Feb 18, 2010 5:14 PM (in response to mcking)
    Re: Questions on the Zenoss Appliance

    Honestly, you have no idea whether any tweaking was done to the Appliance.  It's not a clean install, it's a ready to go install with who knows what changes already done.  The way you're describing it, you'd think that the appliance were "magical" or something, but that's not the case.  If a clean install of Linux with a clean install of Zenoss was outperformed by the appliance, then it's obviously because of some configuration differences on the appliance.  When I said that it's not good for production I wasn't referring to the performance.  I was referring to the fact that people who use the appliance always run in to all sorts of crazy weird issues that no one runs in to when not using the appliance.  Do a fresh install, do the tweaks that I pasted, and see how it goes.

     

    If you had bad luck with the CentOS/RPM install then try Ubuntu Server with the Zenoss Stack Installer.  That's what I'm using and I'm monitoring a ton of devices on my Zenoss servers with no performance issues.

     

    I'll also point out that I helped someone out in the Zenoss chatroom the other day whose Zenoss pages were taking like 2 minutes each to load.  He was monitoring a large number of devices, similar to yourself.  I had him run through the tweaks listed here: docs/DOC-2521.  After he ran through the tweaks he described it as "screaming fast".

  • jmp242 ZenossMaster
    jmp242
    4,060 posts since
    Mar 7, 2007
    Currently Being Moderated
    6. Feb 19, 2010 8:46 AM (in response to Ryan Matte)
    Re: Questions on the Zenoss Appliance

    I want to chime in and agree with Ryan entirely. Users on the Appliance

    for any amount of production use always seem to have issues. Often, they

    end up re-installing on CENTOS or RHEL. In fact, Matt Ray (Zenoss

    Employee) has hinted that around the 2.6 release timeframe, Zenoss Inc

    is planning on changing the appliance to run on CENTOS.

     

    I haven't run CENTOS + Zenoss on a VM, but it your experience is not

    typical of other users. I don't know why you're getting such bad

    performance, but certainly tuning the CENTOS install would be a good

    place to start.

    --

    James Pulver

    Information Technology Area Supervisor

    LEPP Computer Group

    Cornell University

     

     

     

    Ryan Matte wrote, On 2/18/2010 5:11 PM:

    Honestly, you have no idea whether any tweaking was done to the Appliance.  It's not a clean install, it's a ready to go install with who knows what changes already done.  The way you're describing it, you'd think that the appliance were "magical" or something, but that's not the case.  If a clean install of Linux with a clean install of Zenoss was outperformed by the appliance, then it's obviously because of some configuration differences on the appliance.  When I said that it's not good for production I wasn't referring to the performance.  I was referring to the fact that people who use the appliance always run in to all sorts of crazy weird issues that no one runs in to when not using the appliance.  Do a fresh install, do the tweaks that I pasted, and see how it goes.

    >

  • mcking Newbie
    mcking
    5 posts since
    Feb 16, 2010
    Currently Being Moderated
    7. Feb 19, 2010 9:47 AM (in response to jmp242)
    Re: Questions on the Zenoss Appliance

    OK, I really didn't intend for this to be a discussion on whether or not to use the appliance for production (but if no one is supposed to use the appliance, then why make it available for use?).  As far as I can tell, the appliance is the exact same thing as installing a basic barebones install of Linux and then installing Zenoss into it.  No tuning of the OS or Zenoss, just a basic install (I checked the Zenoss config, everything is the default settings out of the box on the appliance).

     

    Obviously my message is not getting across about the tuning that we did on the CentOS VM.  I spent an entire week tuning everything from the filesystem scheduler to the networking stack and I was only able to get the system from a consistent 12+ load and consistent 80% waitio (with 600 devices only doing ping up/down monitoring and a dozen doing a full snmp scan of hundreds of datapoints) to a consistent load of around 4 with consistent waitio in the 30-50% range, consistent 30+ second login times via SSH, and consistent timeouts in the web interface and "hung" processes for zenhub and zensnmpperf.  I don't know what magic fairy dust happened with the appliance, but we are able to monitor all 600 devices with full snmp monitoring and 208000 data points.

     

    At this point I am at the mercy of our Network team.  They have already spent days getting their devices setup the way they want on the appliance and I am loathe to approach them with a fresh install.  They already had a switch go out last week and had no notification because we were still working on getting the system stable, much less turning on notifications!

     

    Thanks for all the advice, I will go back to them with what I have and see what they say....

  • Andrea Consadori ZenossMaster
    Andrea Consadori
    863 posts since
    Feb 11, 2008
    Currently Being Moderated
    8. Feb 19, 2010 10:15 AM (in response to mcking)
    Re: Questions on the Zenoss Appliance

    Hi,

    i told you to make a centos fresh install only because vm is based on centos so a migration will be very easy using zenbackup and zenrestore.

  • Ryan Matte ZenossMaster
    Ryan Matte
    653 posts since
    Mar 26, 2009
    Currently Being Moderated
    9. Feb 19, 2010 10:22 AM (in response to mcking)
    Re: Questions on the Zenoss Appliance

    Yeh, you could use zenbackup and zenrestore to import everything over from the appliance to the fresh install and test with it.  Maybe you can run both until you get a fresh install working properly and then make the switch over.  If that's not possible then have fun with the appliance.

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points