mgcarrete:

As you noted you can increase the queue length. All this length is is a buffer before events are simply discarded. High event queues can occur when daemons first start up, if daemons are reconfiguring too often for the load they carry or if there is a severe bottle neck at zenhub or your mysqld. Can you post the collector's reconfiguration period? Can you also post the load on the collector, number of devices, type of devices, collection interval and hardware (cpu, ram, disk platform) of the collector?

Best,

--Hackman238

mgcarrete:

You're monitoring 2000 processes over 3 devices? As in like 650 processes on each? I just want to be sure I understand.

Best,

--Hackman238

mgcarrete:

I'm not positive that's going to work. That's a pretty special situation. Does your disk IO look like its a bottle neck?

To the zenprocess.conf:

cacheconfigs true

maxqueuelen 500000

eventflushchunksize 200

To the zenhub.conf (the hub that zenprocess is connecting to in the event of multi collectors, etc):

workers 6

cachesize 100000

pcachesize 2000

Restart the zenhub then restart zenprocess. Let me know how it turns out.

Best,--Hackman238

mgcarrete:

Sorry, I was under the impression you had 16 processors since you said 15 Xeon's (figured 0-15). I'd back the workers off to 3. Taking a quick peek, you're right, zenprocess doesnt have caching in v2/v3.  Anyway, if the event queue is being trimed its because the zenhub is being flooded. When the queue exceeds the limit its trimmed and some events might be discarded. This often leads to lost clear events which would be why some of your events persist. Zenhub becomes very easily flooded. Zenhub can become flooded if its constantly reconfiguring large batches, eventchunking is too large, caching is too small, has too few workers, or if mysqld is badly backed up.

Best,

--Hackman238