Same issue here,

I have downloaded zenoss 4.2 as a VMware player and the OS that are used are CentOS 5.4 so maybe that is the version we should use.

[root@localhost ~]# cat /etc/redhat-release

  CentOS release 5.4 (Final

Same problem on ScientificLinux 6.2 - 6.3, CentOS 6.3 only on xen domU, tried on KVM - all ok.

After install zenhub running but with enabled debug there are messages about all workers busy and job queue grows.

OS:

[root@zenoss ~]# cat /etc/redhat-release

Scientific Linux release 6.2 (Carbon)

[root@zenoss ~]# uname -a

Linux zenoss 2.6.32-220.el6.x86_64 #1 SMP Sat Dec 10 17:04:11 CST 2011 x86_64 x86_64 x86_64 GNU/Linux

zenprocess.log:

2012-10-10 11:21:42,234 INFO zen.zenprocess: Connecting to localhost:8789

2012-10-10 11:22:12,239 ERROR zen.zenprocess: Timeout connecting to zenhub: is it running?

2012-10-10 11:22:12,240 INFO zen.maintenance: Performing periodic maintenance

2012-10-10 11:22:12,242 INFO zen.zenprocess: 0 devices processed (0 datapoints)

2012-10-10 11:22:12,242 INFO zen.collector.scheduler: Tasks: 1 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0

2012-10-10 11:22:12,243 WARNING zen.zenprocess: No service named 'EventService': ZenHub may be disconnected

2012-10-10 11:22:12,243 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down

2012-10-10 11:22:12,244 ERROR zen.collector.config: Task configLoader configure failed: ZenHub is down

zenhub.log (with debug):

2012-10-10 11:23:06,585 DEBUG zen.hub: adding listener for localhost:EventService

2012-10-10 11:23:06,587 DEBUG zen.hub: adding listener for localhost:PingPerformanceConfig

2012-10-10 11:23:06,596 DEBUG zen.ZenHub: worklist has 1 items

2012-10-10 11:23:06,596 DEBUG zen.ZenHub: get candidate workers for sendEvents...

2012-10-10 11:23:06,596 DEBUG zen.ZenHub: candidate workers are [1]

2012-10-10 11:23:06,596 DEBUG zen.ZenHub: Giving sendEvents to worker 1, (localhost:Products.ZenHub.services.EventService.sendEvents)

2012-10-10 11:23:06,634 DEBUG zen.ZenHub: worklist has 1 items

2012-10-10 11:23:06,634 DEBUG zen.ZenHub: all workers are busy

2012-10-10 11:23:07,033 DEBUG zen.ZenHub: worklist has 1 items

2012-10-10 11:23:07,033 DEBUG zen.ZenHub: all workers are busy

2012-10-10 11:23:07,974 DEBUG zen.hub: adding listener for localhost:EventService

2012-10-10 11:23:07,977 DEBUG zen.hub: adding listener for localhost:ZenStatusConfig

2012-10-10 11:23:07,986 DEBUG zen.ZenHub: worklist has 2 items

2012-10-10 11:23:07,986 DEBUG zen.ZenHub: all workers are busy

2012-10-10 11:23:07,986 DEBUG zen.ZenHub: worklist has 3 items

2012-10-10 11:23:07,987 DEBUG zen.ZenHub: all workers are busy

2012-10-10 11:23:08,025 DEBUG zen.ZenHub: worklist has 4 items

2012-10-10 11:23:08,025 DEBUG zen.ZenHub: all workers are busy

2012-10-10 11:23:12,033 DEBUG zen.ZenHub: worklist has 4 items

2012-10-10 11:23:12,035 DEBUG zen.ZenHub: all workers are busy

2012-10-10 11:23:12,057 DEBUG zen.Events: ===============  incoming event  ===============

2012-10-10 11:23:12,057 DEBUG zen.Events: Got a localhost zenhub heartbeat event (timeout 90 sec).

2012-10-10 11:23:12,057 DEBUG zen.zenoss.protocols.amqp: Publishing with routing key zenoss.heartbeat.localhost to exchange zenoss.heartbeats

2012-10-10 11:23:17,035 DEBUG zen.ZenHub: worklist has 4 items

2012-10-10 11:23:17,036 DEBUG zen.ZenHub: all workers are busy

2012-10-10 11:23:22,036 DEBUG zen.ZenHub: worklist has 4 items

2012-10-10 11:23:22,037 DEBUG zen.ZenHub: all workers are busy

2012-10-10 11:23:23,854 DEBUG zen.hub: adding listener for localhost:EventService

2012-10-10 11:23:23,863 DEBUG zen.hub: adding listener for localhost:ModelerService

2012-10-10 11:23:23,868 DEBUG zen.ZenHub: worklist has 5 items

2012-10-10 11:23:23,868 DEBUG zen.ZenHub: all workers are busy

2012-10-10 11:23:23,870 DEBUG zen.ZenHub: worklist has 6 items

2012-10-10 11:23:23,870 DEBUG zen.ZenHub: all workers are busy

2012-10-10 11:23:25,751 DEBUG zen.ZenHub: worklist has 7 items

2012-10-10 11:23:25,751 DEBUG zen.ZenHub: all workers are busy

2012-10-10 11:23:27,038 DEBUG zen.ZenHub: worklist has 7 items

2012-10-10 11:23:27,039 DEBUG zen.ZenHub: all workers are busy

Strace of zenhub process in attachment. VM running and i can provide additional info.

Default installation of zenoss 4.2, no monitored devices, default configs.

Xen domU, 4 VCPU, 2G RAM, 20Gb FC (Storwise 7000).

Only localhost as monitored device but it is not modeled.

zeneventserver.conf:

# The URI to the RabbitMQ server. This property is ignored if

# $ZENHOME/etc/global.conf can be read in which the AMQP URI will be

# built from the properties found in that location.

#zep.amqp.uri=amqp://zenoss:zenoss@localhost:5672/zenoss

# The number of milliseconds to wait between connection retries

#zep.amqp.retry=60000

# The default maximum number of events returned in an event query

#zep.query.limit=1000

# The maximum number of minutes a closed event may not be seen again before moving to archive.

#zep.max_event_archive_interval_minutes=43200

# JDBC Protocol

# Loaded from global.conf zep_db_type first, then falls back to the definition in this file.

#zep.jdbc.protocol=mysql

# JDBC Hostname

# Loaded from global.conf zep_host first, then falls back to the definition in this file.

#zep.jdbc.hostname=localhost

# JDBC Port

# Loaded from global.conf zep_port first, then falls back to the definition in this file.

#zep.jdbc.port=3306

# JDBC Database Name

# Loaded from global.conf zep_db first, then falls back to the definition in this file.

#zep.jdbc.dbname=zenoss_zep

# JDBC Username

# Loaded from global.conf zep_user first, then falls back to the definition in this file.

#zep.jdbc.username=zenoss

# JDBC Password

# Loaded from global.conf zep_password first, then falls back to the definition in this file.

#zep.jdbc.password=zenoss

# The default auto commit state of pooled connections

#zep.jdbc.pool.default_auto_commit=false

# The default read only state of pooled connections

#zep.jdbc.pool.default_read_only=false

# The default transaction isolation of pooled connections

#zep.jdbc.pool.default_transaction_isolation=2

# The maximum number of active connections in the pool

#zep.jdbc.pool.max_active=10

# The maximum number of idle connections in the pool

#zep.jdbc.pool.max_idle=10

# The minimum number of idle connections in the pool

#zep.jdbc.pool.min_idle=3

# The initial number of connections in the pool

#zep.jdbc.pool.initial_size=3

# The maximum amount of time to wait for a connection from the pool

#zep.jdbc.pool.max_wait=-1

# Whether to validate connections when borrowing from the pool

#zep.jdbc.pool.test_on_borrow=false

# Whether to validate connections when returning to the pool

#zep.jdbc.pool.test_on_return=false

# Whether to validation connections periodically while idle

#zep.jdbc.pool.test_while_idle=false

# Amount of time between idle connection validation, abandoned cleaner, and

# idle pool resizing.

#zep.jdbc.pool.time_between_eviction_runs_millis=5000

# Amount of time before a connection is considered idle

#zep.jdbc.pool.min_evictable_idle_time_millis=60000

# Interceptors to run

#zep.jdbc.pool.jdbc_interceptors=ConnectionState

# How often to validate connections in the pool

#zep.jdbc.pool.validation_interval=30000

# Whether JMX can be used to configure the pool

#zep.jdbc.pool.jmx_enabled=true

# True if a fair queue is used by the connection pool.

#zep.jdbc.pool.fair_queue=true

# Timed out connections will get closed when this percentage of the pool is full.

#zep.jdbc.pool.abandon_when_percentage_full=0

# Time in milliseconds to keep connections alive.

#zep.jdbc.pool.max_age=0

# Set to true if ProxyConnection class should use String.equals instead of ==

# when comparing method names.

#zep.jdbc.pool.use_equals=true

# Whether to remove abandoned connections if they exceed the

# remove_abandoned_timeout.

#zep.jdbc.pool.remove_abandoned=false

# The amount of time (in seconds) before a connection is considered abandoned.

#zep.jdbc.pool.remove_abandoned_timeout=60

# Whether to log stack traces of abandoned connections.

#zep.jdbc.pool.log_abandoned=false

# Instead of abandoning connections, logs a warning if log_abandoned is true.

# If this value is <= 0, then no suspect checking will be performed.

#zep.jdbc.pool.suspect_timeout=0

# The default number of retries performed when a database deadlock is encountered.

#zep.jdbc.deadlock_retries=5

# The core pool size used by the primary executor service.

# By default, this is set to the number of processors in the system.

#zep.executor.core_pool_size=4

# The maximum pool size used by the primary executor service.

# By default, this is set to the number of processors in the system.

#zep.executor.max_pool_size=4

# The maximum number of entries waiting to execute on the executor service.

#zep.executor.queue_capacity=10

# Index Directory

#zep.index.dir=var/zeneventserver/index

# Index RAM buffer size

#zep.index.ram_buffer_size_mb=16.0

# Heartbeat check interval in seconds

#zep.heartbeat.interval_seconds=60

# Database optimization is performed on this interval (in minutes). Specify an interval <= 0 to disable.

#zep.database.optimize_minutes=60

## Comma separated list of ZEP plug-ins to disable.

#zep.plugins.disabled=

## Whether to disable all external ZEP plug-ins (from ZenPacks).

#zep.plugins.external.disabled=false

## Optional configuration settings for each plug-in

#plugin.<ID>.<NAME>=<VALUE>

## The default number of trigger rule compilations to cache.

#plugin.TriggerPlugin.triggerRuleCacheSize=200

## Partitioning parameters for partitioned tables in ZEP.

##

## Values are:

##

##   partition.<table>.unit = The TimeUnit of each partition.

##   partition.<table>.duration = The duration of each partition.

##   partition.<table>.initial_past_partitions = The initial number of partitions to create in the past.

##   partition.<table>.future_partitions = The number of future partitions to maintain.

##

#partition.event_archive.unit=DAYS

#partition.event_archive.duration=1

# We want to configure the maximum number of initial partitions for migration.

#partition.event_archive.initial_past_partitions=90

#partition.event_archive.future_partitions=3

#partition.event_time.unit=HOURS

#partition.event_time.duration=1

# We want to configure the maximum number of initial partitions for migration.

#partition.event_time.initial_past_partitions=0

#partition.event_time.future_partitions=2

zenhub.conf:

#

# Configuration file for zenhub

#

#  To enable a particular option, uncomment the desired entry.

#

# Parameter     Setting

# ---------     -------

#

# Logging severity threshold, default: 20

#logseverity 20

#

# Override the default logging path

#logpath None

#

# Max size of log file in KB; default

#  10240, default: 10240

#maxlogsize 10240

#

# Max number of back up log files; default

#  3, default: 3

#maxbackuplogs 3

#

# Use an alternate configuration file

#configfile None

#

# Generate an XML file containing command-line

#  switches., default: False

#genxmlconfigs False

#

# User to become when running default:zenoss,

#  default: zenoss

#uid zenoss

#

# Cycle continuously on cycleInterval

#  from Zope, default: False

#cycle False

#

# Launch into the background, default: False

#daemon False

#

# Log to console and log file, default: False

#duallog False

#

# output log info in HTML table format,

#  default: False

#weblog False

#

# Run under a supervisor which will restart

#  it, default: False

#watchdog False

#

# The path to the watchdog reporting socket

#watchdogPath None

#

# Wait seconds for initial heartbeat

#starttimeout None

#

# Set listener socket options.For option

#  details: man 7 socket, default: []

#socketOption []

#

# Port to use for XML-based Remote Procedure

#  Calls (RPC), default: 8081

#xmlrpcport 8081

#

# Port to use for Twisted's pb service,

#  default: 8789

#pbport 8789

#

# File where passwords are stored, default:

#  /Users/pkw/Projects/zenoss/trunk/home/etc/hubpasswd

#passwd /Users/pkw/Projects/zenoss/trunk/home/etc/hubpasswd

#

# Name of the distributed monitor this

#  hub runs on, default: localhost

#monitor localhost

#

# Number of worker instances to handle

#  requests, default: 2

#workers 2

#

# Run higher priority jobs before lower

#  priority ones, default: False

#prioritize False

#

# Allow any priority job to run on any

#  worker, default: False

#anyworker False

#

# Log current worker state to $ZENHOME/log/workerstats,

#  default: False

#logworkerstats False

#

# root object for data load (i.e. /zport/dmd),

#  default: /zport/dmd

#zodb-dataroot /zport/dmd

#

# in memory cachesize default: 1000, default:

#  1000

#zodb-cachesize 1000

#

# hostname of the MySQL server for ZODB,

#  default: localhost

#zodb-host localhost

#

# port of the MySQL server for ZODB, default:

#  3306

#zodb-port 3306

#

# user of the MySQL server for ZODB, default:

#  zenoss

#zodb-user zenoss

#

# password of the MySQL server for ZODB,

#  default: zenoss

#zodb-password zenoss

#

# Name of database for MySQL object store,

#  default: zodb

#zodb-db zodb

#

# Name of socket file for MySQL server

#  connection if host is localhost

#zodb-socket None

#

# memcached servers to use for object

#  cache (eg. 127.0.0.1:11211), default:

#  127.0.0.1:11211

#zodb-cacheservers 127.0.0.1:11211

#

# Defer polling the database for the specified

#  maximum time interval, in seconds.

#  This will default to 60 only if --zodb-cacheservers

#  is set.

#zodb-poll-interval None

#

zeneventd.conf:

#

# Configuration file for zeneventd

#

#  To enable a particular option, uncomment the desired entry.

#

# Parameter     Setting

# ---------     -------

#

# Logging severity threshold, default: 20

#logseverity 20

#

# Override the default logging path

#logpath None

#

# Max size of log file in KB; default

#  10240, default: 10240

#maxlogsize 10240

#

# Max number of back up log files; default

#  3, default: 3

#maxbackuplogs 3

#

# Use an alternate configuration file

#configfile None

#

# Generate an XML file containing command-line

#  switches., default: False

#genxmlconfigs False

#

# User to become when running default:zenoss,

#  default: zenoss

#uid zenoss

#

# Cycle continuously on cycleInterval

#  from Zope, default: False

#cycle False

#

# Launch into the background, default: False

#daemon False

#

# Log to console and log file, default: False

#duallog False

#

# output log info in HTML table format,

#  default: False

#weblog False

#

# Run under a supervisor which will restart

#  it, default: False

#watchdog False

#

# The path to the watchdog reporting socket

#watchdogPath None

#

# Wait seconds for initial heartbeat

#starttimeout None

#

# Set listener socket options.For option

#  details: man 7 socket, default: []

#socketOption []

#

# Cycle, in seconds, for maintenance tasks.

#  Default is 60., default: 60

#maintenancecycle 60

#

# Force sync() before processing every

#  event; default is to sync() no more

#  often than once every 1/2 second.,

#  default: False

#synceveryevent False

#

# root object for data load (i.e. /zport/dmd),

#  default: /zport/dmd

#zodb-dataroot /zport/dmd

#

# in memory cachesize default: 1000, default:

#  1000

#zodb-cachesize 1000

#

# hostname of the MySQL server for ZODB,

#  default: localhost

#zodb-host localhost

#

# port of the MySQL server for ZODB, default:

#  3306

#zodb-port 3306

#

# user of the MySQL server for ZODB, default:

#  zenoss

#zodb-user zenoss

#

# password of the MySQL server for ZODB,

#  default: zenoss

#zodb-password zenoss

#

# Name of database for MySQL object store,

#  default: zodb

#zodb-db zodb

#

# Name of socket file for MySQL server

#  connection if host is localhost

#zodb-socket None

#

# memcached servers to use for object

#  cache (eg. 127.0.0.1:11211), default:

#  127.0.0.1:11211

#zodb-cacheservers 127.0.0.1:11211

#

# Defer polling the database for the specified

#  maximum time interval, in seconds.

#  This will default to 60 only if --zodb-cacheservers

#  is set.

#zodb-poll-interval None

#

I do not using distributed collectors or something else - i am just trying to install it with basic setup =)))

Nothing there...

[root@zenoss ~]# /usr/sbin/rabbitmqctl list_queues -p /zenoss

Listing queues ...

celery          0

zenoss.queues.zep.modelchange          0

zenoss.queues.zep.signal          0

zenoss.queues.zep.migrated.summary          0

zenoss.queues.zep.rawevents          0

zenoss.queues.zep.heartbeats          0

zenoss.queues.zep.zenevents          0

zenoss.celeryd.pidbox          0

zenoss.queues.zep.migrated.archive          0

...done.

I am having a similar issue with ZenHub.  Has this issue been addressed with 4.2?

I had the same problem with my 4.2 on rhel6.

My problem was the rabbitmq-server. The rabbitmq-server blocks by default every connection, if the memory is over 40% or the filesystem hasn't min. 1gb free space.

Check the following as root:

rabbitmqctl list_connections

If there is something with "blocked" or "blocking" you have the same problem.

You can modify this limitations:

http://www.rabbitmq.com/memory.html