joan,

The best options short of creating a second logical collector on the same physical collector is to change the testing approach.

A short term option would be to optimize the code of the script zencommand is calling at scale. This can add up drmatically. Chaining sed, awk and other bins is very common, but each one takes up precious ms and causes thread contention. Every optimization you can make counts.

A better option would be to use PythonCollector (https://github.com/Hackman238/ZenPacks.zenoss.PythonCollector) as it's quite a bit more scalable than zencommand. The downside is your test would need to be rewritten in python.

The best option, in my opinion, is to try to use a similar test that fits under other zendaemons (like SNMP) or to craft a specialized daemon dedicated to the test needing large scale. This is usually isolated to cases where you're needing very large scale, producing a package for a customer for which scale is unknown or needing to create a custom datasource.

The big problem with zencommand isn't really zencommand at all- it's the nature of what it's designed to do, execute scripts and commands. The competition for execution resources really adds up fast, especially with high parallel values. What's worse is the collectors are often subjec to high IO wait from RRD writting to the disks. With that it's very easy for a normally fast script to dramatically bloat in execution time waiting for disk access.

--Shane Scott (Hackman238)

I'm not Shane, but he means going into the  GUI on Zenoss and (at least in 3.2.x) going to Advanced -> Collectors, and adding another one with the gear icon... This can increase parallelism of the checks, if your server has the hardware to not bog down (and you manually load balance devices across the 2 collectors)...

--

James Pulver

ZCA Member

LEPP Computer Group

Cornell University

I don't think you have to do anything else - you then would just move devices to the new collector. . . Then I think it will start up additional daemons.

--

James Pulver

ZCA Member

LEPP Computer Group

Cornell University

Joan -

Setting up logical collectors isn't that hard but it a little bit of manual work.  The steps are as follows:

1) Create the collector in the zenoss UI (let's say you call it collector1)

2) On the Zenoss server itself you have to duplicate all the daemons you want to run on this collector.  So you would copy $ZENHOME/bin/zencommand to $ZENHOME/bin/collector1_zencommand, $ZENHOME/bin/zenperfsnmp to $ZENHOME/bin/collector1_zenperfsnmp, etc.

3) You then have to modify those files.  So modify collector1_zencommand and set CFGFILE=$CFGDIR/collector1_zencommand.conf

4) For each of your collector1_ daemons you need a conf file in $ZENHOME/etc.  For example $ZENHOME/etc/collector1_zencommand.conf (this file name matches what you set CFGFILE to).  In the conffile you set the 'monitor' value to be what you configured in the UI (in my example 'collector1').  At a minimum you need:

monitor     collector1

hubhost     localhost

This is assuming you only have one zenoss server so the hub is the same machine.

You can then have all the other options that the daemons can have (in the case of zencommand; parallel etc.)

Then you have to create a $ZENHOME/etc/daemons.txt and in this file you list all of the collector1_ daemons you want to start.  This should start all the normal zenoss deamons and the collector1_ daemons when you do a 'zenoss start'

Then when you assign a device to the 'collector1' collector it is these collector1_ daemons that are monitoring it

--Dennis