Hello all.

I have an old 3.2.1 instance that I have been toying around with for a while, testing Zenoss and monitoring a few devices (35-40 servers). Recently I was ready to move into production state, and wanted to upgrade to the latest Zenoss version. I decided to do a fresh start to get away from the old stack installer that I was using, and wasn't planning on migrating any settings - the old system was ready to be ditched completely. Hence, yesterday I installed a CentOS 6.3 basic server and went through the autodeploy-process. Within minutes, my shiny, new Zenoss Core 4.2.0 instance was up and running.

For a couple of hours, I hardened CentOS a bit (enabling firewall, tightening ssh) and started adding devices to Zenoss. Everything was working perfectly, but then I noticed that the system clock was two hours off, and I startet NTP to get an accurate clock. I also renamed the server (as it was moved from preprod to production). I am not sure if this is the reason for the problems that followed, but thought I should mention it.

Now, after a system reboot and/or Zenoss restart, zenhub appears to be starting, then dies. All other Zenoss processes start as expected. In the zenhub.log, I get the following entries:

2012-10-16 10:43:45,330 INFO zen.HubService.RenderConfig: Starting graph retrieval listener on port 8090

2012-10-16 10:43:48,342 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

2012-10-16 10:43:51,346 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

2012-10-16 10:43:51,346 ERROR zen.ZenHub: Unable to send an event

Traceback (most recent call last):

  File "/opt/zenoss/Products/ZenHub/zenhub.py", line 576, in sendEvent

    self.zem.sendEvent(Event(**kw))

  File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 60, in sendEvent

    event = self._publishEvent(event)

  File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 82, in _publishEvent

    publisher.publish(event)

  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 283, in publish

    self._publish("$RawZenEvents", routing_key, event, mandatory=mandatory, immediate=immediate)

  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 302, in _publish

    mandatory, immediate)

  File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 376, in publish

    headers=headers, declareExchange=declareExchange)

  File "/opt/zenoss/lib/python/zenoss/protocols/amqp.py", line 138, in publish

    raise Exception("Could not publish message. Connection may be down")

Exception: Could not publish message. Connection may be down

2012-10-16 10:43:57,255 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

2012-10-16 10:44:00,259 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

I can log in to the web GUI, and I can see all my devices, but no errors are logged, and no graphs are graphed. I can no longer add anything to the system (no users or devices), and no changes to the devices/groups/systems/locations hierarchy can be made. Basically, I have a shiny new system that tells me that nothing is wrong as no events are received or handled, but clearly something is very wrong.

Does Zenoss store the hostname and/or IP address in any of it's config files or the database, causing these problems when changing name/IP?

Does any of you know where I should start looking, to avoid spending hours looking in the wrong place? Instead of spending a workday or two trying blindly to fix the problem, I might as well reinstall CentOS and Zenoss, but that wouldn't teach me anything and wouldn't be any fun.

Thanks in advance!