Archived community.zenoss.org | full text search
Skip navigation
508 Views 1 Reply Latest reply: Feb 18, 2014 6:52 AM by Robert Light RSS
Robert Light Rank: White Belt 11 posts since
Oct 9, 2012
Currently Being Moderated

Feb 3, 2014 8:53 AM

Infrastructure in tact however no events appearing

Hi All!


I would like to believe that our infrastructure is that stable but i know that isnt the case!

 

We are using Zenoss 4.2.0

 

Scenario:

Nothing has appeared to of changed in the environment but all of a sudden we are not getting any events for any of our devices and we have noticed that on the odd occasion one or two of the demons would stop and would need restarting (not unusual for us though).


I am a windows admin so please be gentle!

 

Looking at the demons and their associated logs i can see that we have problems but need a starting piont. Some errors:

 

zenactiond

2014-02-03 12:58:12,678 INFO zen.maintenance: Performing periodic maintenance

2014-02-03 12:58:12,679 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 111] Connection refused

2014-02-03 12:58:12,680 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 111] Connection refused

2014-02-03 12:58:12,682 ERROR zen.maintenance: Maintenance failed. Message from hub: Could not publish message. Connection may be down

2014-02-03 12:59:12,683 INFO zen.maintenance: Performing periodic maintenance

2014-02-03 12:59:12,684 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 111] Connection refused

2014-02-03 12:59:12,684 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 111] Connection refused

2014-02-03 12:59:12,685 ERROR zen.maintenance: Maintenance failed. Message from hub: Could not publish message. Connection may be down

 

zeneventd

2014-02-03 12:57:41,361 ERROR zen.maintenance: Maintenance failed. Message from hub: Could not publish message. Connection may be down

2014-02-03 12:58:41,368 ERROR zen.maintenance: Maintenance failed. Message from hub: Could not publish message. Connection may be down

2014-02-03 12:59:41,377 ERROR zen.maintenance: Maintenance failed. Message from hub: Could not publish message. Connection may be down

2014-02-03 13:00:41,387 ERROR zen.maintenance: Maintenance failed. Message from hub: Could not publish message. Connection may be down

2014-02-03 13:01:41,394 ERROR zen.maintenance: Maintenance failed. Message from hub: Could not publish message. Connection may be down

 

zeneventlog

2014-02-03 13:01:49,939 INFO zen.maintenance: Performing periodic maintenance

2014-02-03 13:01:49,940 INFO zen.zeneventlog: Counter discardedEvents, value 148146279

2014-02-03 13:01:49,940 INFO zen.zeneventlog: Counter eventCount, value 177093947

2014-02-03 13:01:49,941 INFO zen.zeneventlog: 30 devices processed (0 datapoints)

2014-02-03 13:01:49,943 INFO zen.collector.scheduler: Tasks: 31 Successful_Runs: 4257 Failed_Runs: 0 Missed_Runs: 15 Queued_Tasks: 0 Running_Tasks: 1 

2014-02-03 13:01:50,067 ERROR zen.zeneventlog: Discarding oldest 51 events because maxqueuelen was exceeded: 5051/5000

2014-02-03 13:01:50,130 ERROR zen.zeneventlog: Discarding oldest 51 events because maxqueuelen was exceeded: 5051/5000

2014-02-03 13:02:07,906 ERROR zen.zeneventlog: Discarding oldest 51 events because maxqueuelen was exceeded: 5051/5000

2014-02-03 13:02:07,969 ERROR zen.zeneventlog: Discarding oldest 51 events because maxqueuelen was exceeded: 5051/5000

2014-02-03 13:02:08,140 ERROR zen.zeneventlog: Discarding oldest 51 events because maxqueuelen was exceeded: 5051/5000

2014-02-03 13:02:08,289 ERROR zen.zeneventlog: Discarding oldest 51 events because maxqueuelen was exceeded: 5051/5000

2014-02-03 13:02:08,486 ERROR zen.zeneventlog: Discarding oldest 51 events because maxqueuelen was exceeded: 5051/5000

2014-02-03 13:02:08,543 ERROR zen.zeneventlog: Discarding oldest 51 events because maxqueuelen was exceeded: 5051/5000

 

zenhub

2014-02-03 13:07:06,463 INFO zen.ZenHub: Worker (19051) reports

2014-02-03 13:07:06,462 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 111] Connection refused

2014-02-03 13:07:06,463 INFO zen.ZenHub: Worker (19051) reports

2014-02-03 13:07:06,463 INFO zen.zenoss.protocols.amqp: amqp connection was closed [Errno 111] Connection refused

2014-02-03 13:07:06,463 INFO zen.ZenHub: Worker (19051) reports

2014-02-03 13:07:06,463 ERROR zen.Events: Could not publish message. Connection may be down Traceback (most recent call last): File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 42, in sendEvents self._publishEvent(event, publisher) File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 82, in _publishEvent publisher.publish(event) File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 283, in publish self._publish("$RawZenEvents", routing_key, event, mandatory=mandatory, immediate=immediate) File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 302, in _publish mandatory, immediate) File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 376, in publish headers=headers, declareExchange=declareExchange) File "/opt/zenoss/lib/python/zenoss/protocols/amqp.py", line 138, in publish raise Exception("Could not publish message. Connection may be down") Exception: Could not publish message. Connection may be down

 

zenjobs

Traceback (most recent call last): File "/opt/zenoss/lib/python/celery/worker/consumer.py", line 349, in start self.reset_connection() File "/opt/zenoss/lib/python/celery/worker/consumer.py", line 592, in reset_connection self.connection = self._open_connection() File "/opt/zenoss/lib/python/celery/worker/consumer.py", line 657, in _open_connection self.app.conf.BROKER_CONNECTION_MAX_RETRIES) File "/opt/zenoss/lib/python/kombu/connection.py", line 223, in ensure_connection interval_start, interval_step, interval_max) File "/opt/zenoss/lib/python/kombu/utils/__init__.py", line 158, in retry_over_time return fun(*args, **kwargs) File "/opt/zenoss/lib/python/kombu/connection.py", line 146, in connect return self.connection File "/opt/zenoss/lib/python/kombu/connection.py", line 574, in connection self._connection = self._establish_connection() File "/opt/zenoss/lib/python/kombu/connection.py", line 533, in _establish_connection conn = self.transport.establish_connection() File "/opt/zenoss/lib/python/kombu/transport/amqplib.py", line 278, in establish_connection connect_timeout=conninfo.connect_timeout) File "/opt/zenoss/lib/python/kombu/transport/amqplib.py", line 88, in __init__ super(Connection, self).__init__(*args, **kwargs) File "/opt/zenoss/lib/python/amqplib/client_0_8/connection.py", line 129, in __init__ self.transport = create_transport(host, connect_timeout, ssl) File "/opt/zenoss/lib/python/amqplib/client_0_8/transport.py", line 281, in create_transport return TCPTransport(host, connect_timeout) File "/opt/zenoss/lib/python/amqplib/client_0_8/transport.py", line 85, in __init__ raise socket.error, msg error: [Errno 111] Connection refused

2014-02-03 13:08:25,216 ERROR celery.worker.consumer: Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 2 seconds...

2014-02-03 13:08:27,219 ERROR celery.worker.consumer: Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 4 seconds...

2014-02-03 13:08:31,224 ERROR celery.worker.consumer: Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 6 seconds...

2014-02-03 13:08:37,231 ERROR celery.worker.consumer: Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 8 seconds...

2014-02-03 13:08:45,240 ERROR celery.worker.consumer: Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 10 seconds...

2014-02-03 13:08:55,251 ERROR celery.worker.consumer: Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 12 seconds...

2014-02-03 13:09:07,264 ERROR celery.worker.consumer: Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 14 seconds...

 

All other logs seem to report error free except zopectl which just tells me there is a problem and to forward to zenoss, here is the error details:

 

Type: <type 'exceptions.ValueError'>
Value: 'event' is not a valid daemon name

Traceback (most recent call last): File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 126, in publish request, bind=1) File "/opt/zenoss/lib/python/ZPublisher/mapply.py", line 77, in mapply if debug is not None: return debug(object,args,context) File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 46, in call_object result=apply(object,args) # Type s<cr> to step into published object. File "/opt/zenoss/lib/python/Products/Five/browser/metaconfigure.py", line 476, in __call__ return self.index(*args, **kw) File "/opt/zenoss/lib/python/Products/Five/browser/pagetemplatefile.py", line 125, in __call__ return self.im_func(im_self, *args, **kw) File "/opt/zenoss/Products/ZenWidgets/FileGzipper.py", line 82, in __call__ return original(self, __instance, *args, **keywords) File "/opt/zenoss/lib/python/Products/Five/browser/pagetemplatefile.py", line 59, in __call__ sourceAnnotations=getattr(debug_flags, 'sourceAnnotations', 0), File "/opt/zenoss/lib/python/zope/pagetemplate/pagetemplate.py", line 132, in pt_render strictinsert=0, sourceAnnotations=sourceAnnotations File "/opt/zenoss/lib/python/zope/pagetemplate/pagetemplate.py", line 240, in __call__ interpreter() File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 266, in __call__ self.interpret(self.program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 883, in do_useMacro self.interpret(macro) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal self.do_optTag(stuff) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag return self.no_tag(start, program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag self.interpret(program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 883, in do_useMacro self.interpret(macro) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal self.do_optTag(stuff) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag return self.no_tag(start, program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag self.interpret(program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 883, in do_useMacro self.interpret(macro) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 941, in do_defineSlot self.interpret(slot) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal self.do_optTag(stuff) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag return self.no_tag(start, program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag self.interpret(program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 941, in do_defineSlot self.interpret(slot) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal self.do_optTag(stuff) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag return self.no_tag(start, program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag self.interpret(program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 941, in do_defineSlot self.interpret(slot) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal self.do_optTag(stuff) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag return self.no_tag(start, program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag self.interpret(program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal self.do_optTag(stuff) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag return self.no_tag(start, program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag self.interpret(program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 883, in do_useMacro self.interpret(macro) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal self.do_optTag(stuff) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag return self.no_tag(start, program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag self.interpret(program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 941, in do_defineSlot self.interpret(slot) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal self.do_optTag(stuff) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag return self.no_tag(start, program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag self.interpret(program) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret handlers[opcode](self, args) File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 615, in do_insertText_tal text = self.engine.evaluateText(stuff[0]) File "/opt/zenoss/lib/python/Products/PageTemplates/Expressions.py", line 225, in evaluateText text = self.evaluate(expr) File "/opt/zenoss/lib/python/zope/tales/tales.py", line 696, in evaluate return expression(self) File "/opt/zenoss/lib/python/zope/tales/pythonexpr.py", line 59, in __call__ return eval(self._code, vars) File "<string>", line 1, in <module> File "/opt/zenoss/Products/ZenModel/ZenossInfo.py", line 526, in getLogData filename = self._getLogPath(daemon) File "/opt/zenoss/Products/ZenModel/ZenossInfo.py", line 502, in _getLogPath raise ValueError("%r is not a valid daemon name" % daemon) ValueError: 'event' is not a valid daemon name

 

I am hoping that with the above information someone can give me a starting point. I appologise in advance if this is not in the correct forum.

 

Many thanks

Rob

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points