Archived community.zenoss.org | full text search
Skip navigation
1 2 3 Previous Next 21805 Views 39 Replies Latest reply: Sep 11, 2013 3:08 PM by Emocke RSS
sunnydave Rank: White Belt 26 posts since
May 14, 2007
Currently Being Moderated

Aug 13, 2012 12:55 PM

Can't keep zenhub running

I sure hope someone can help me figure this out.  Been through all the logs, forums, and anything else I can think of, so as a last resort, I'm reaching out the community.  I'm not a power linux user by any means, but  know enough to be dangerous, to give you some background.   All we need is simple monitoring and alerting, but without Zenhub, we don't get alerts, which isn't good.  Zenhub won't stay running for a more than 2-3 days at this point, and it's been that way for months.  There's nothing in the zenhub log.  It just stops.  I found this in zencommand log, which is the most information I can find on the issue.

 

2012-08-10 21:35:52,849 ERROR zen.zencommand: ZenHub is down

Traceback (most recent call last):

  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 704, in doFetchConfig

    self.setPropertyItems(driver.next())

  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result

    raise ex

HubDown: ZenHub is down

2012-08-10 21:35:53,342 ERROR zen.zencommand: ZenHub is down

Traceback (most recent call last):

  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 752, in start

    driver.next()

  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result

    raise ex

HubDown: ZenHub is down

2012-08-11 03:35:53,484 ERROR zen.zencommand: ZenHub is down

Traceback (most recent call last):

  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 704, in doFetchConfig

    self.setPropertyItems(driver.next())

  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result

    raise ex

HubDown: ZenHub is down

2012-08-11 03:35:53,685 ERROR zen.zencommand: ZenHub is down

Traceback (most recent call last):

  File "/usr/local/zenoss/zenoss/Products/ZenRRD/zencommand.py", line 752, in start

    driver.next()

  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Driver.py", line 64, in result

    raise ex

 

We are running this on Ubuntu Linux 12.04.1 and Zenoss is the latest version: Zenoss 3.2.1.

 

I'd truly appreciate some help in getting this fixed.  Please let me know if there is any other information I can provide.  Thank you in advance.

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    1. Aug 18, 2012 5:36 AM (in response to sunnydave)
    Re: Can't keep zenhub running

    sunnydave:

     

    On the zenoss master su zenoss then zenhub stop. Once it's stopped, ps aux | grep zenhub and kill any remaning workers if there are any. Once done, zenhub start -v10. Keep an eye on the $ZENHOME/log/zenhub.log. Post the log when zenhub dies. Can you also please post your $ZENHOME/etc/zenhub.conf?

     

    Thanks!

     

    Best,

    --Shane Scott (Hackman238)

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    3. Aug 23, 2012 7:47 PM (in response to sunnydave)
    Re: Can't keep zenhub running

    sunnydave:

     

    Anytime.

     

    I await the log. Thanks man!

     

    Best,

    --Shane Scott (Hackman238)

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    6. Aug 27, 2012 1:29 PM (in response to sunnydave)
    Re: Can't keep zenhub running

    sunnydave:

     

    Can you email them to me at hackman238@gmail.com?

     

    Best,

    --Shane Scott (Hackman238)

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    8. Aug 27, 2012 4:01 PM (in response to sunnydave)
    Re: Can't keep zenhub running

    Oh no! Somebody dump sacked my inbox!

     

    Just kidding. I got your logs sunnydave. I'll take a look and get back to you.

     

    Best,

    --Shane Scott (Hackman238)

  • cjhudock Newbie 3 posts since
    Aug 28, 2012
    Currently Being Moderated
    9. Aug 28, 2012 9:32 AM (in response to sunnydave)
    Re: Can't keep zenhub running

    Hello Gentlemen,

    First of all I’m a newbie to Zenoss.   I downloaded the 4.2 VM image and loaded on an ESX server. After loading a few devices and a few MIB, I too am seeing my zenhub crash. It will not stay up at all. It immediately shuts down.

     

    When I attempt to restart the service I get the following in the zenhub log:

     

    2012-08-28 09:14:01,227 INFO zen.HubService.RenderConfig: Starting graph retrieval listener on port 8090

    2012-08-28 09:14:04,239 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

    2012-08-28 09:14:07,242 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

    2012-08-28 09:14:07,242 ERROR zen.ZenHub: Unable to send an event

    Traceback (most recent call last):

      File "/opt/zenoss/Products/ZenHub/zenhub.py", line 576, in sendEvent

        self.zem.sendEvent(Event(**kw))

      File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 60, in sendEvent

        event = self._publishEvent(event)

      File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 82, in _publishEvent

        publisher.publish(event)

      File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 283, in publish

        self._publish("$RawZenEvents", routing_key, event, mandatory=mandatory, immediate=immediate)

      File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 302, in _publish

        mandatory, immediate)

      File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 376, in publish

        headers=headers, declareExchange=declareExchange)

      File "/opt/zenoss/lib/python/zenoss/protocols/amqp.py", line 138, in publish

        raise Exception("Could not publish message. Connection may be down")

    Exception: Could not publish message. Connection may be down

    2012-08-28 09:14:12,730 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

    2012-08-28 09:14:15,734 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 104] Connection reset by peer

     

    I deleted all of the zenhubworker processes with the same results.

     

    Any idea?

     

    Chuck

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    10. Aug 28, 2012 10:26 AM (in response to cjhudock)
    Re: Can't keep zenhub running

    Chuck:

     

    Looks like your rabbitmq-server daemon might not be running. Check that and restart zenhub.

     

    Best,

    --Shane Scott (Hackman238)

  • cjhudock Newbie 3 posts since
    Aug 28, 2012
    Currently Being Moderated
    11. Aug 28, 2012 10:51 AM (in response to Shane Scott)
    Re: Can't keep zenhub running

    Shane,

     

     

     

    The rabbitmg-server appears to be running. Performed a restart and checked

    status.

     

     

     

    Deleted zenhubworkers process and performed a start from the

    Zenoss:Daemons. Same result.

     

     

     

    Chuck

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    12. Aug 28, 2012 11:03 AM (in response to cjhudock)
    Re: Can't keep zenhub running

    Chuck:

     

    Interesting. I would double check that mysqld is running as well. Next Double check that $ZENHOMe/etc/globals.conf points to the local rabbit & mysqld instance with valid creds.

     

    Best,

    --Shane Scott (Hackman238)

  • dpetzel Rank: Brown Belt 1,141 posts since
    Oct 17, 2010
    Currently Being Moderated
    14. Aug 29, 2012 6:14 PM (in response to sunnydave)
    Re: Can't keep zenhub running

    This sounds alot like message/67159#67159. Have you looked at creating the rabbitmq-env.conf file?

1 2 3 Previous Next

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points