Archived community.zenoss.org | full text search
Skip navigation
1 2 3 Previous Next 7292 Views 32 Replies Latest reply: Dec 10, 2013 5:24 PM by residentbiscuit RSS
Pavel Perez Newbie 1 posts since
Sep 4, 2012
Currently Being Moderated

Sep 4, 2012 3:26 PM

zepConnectionError timeout service

Hello,

 

I recently installed Zenoss 4.2 in the network I manage, everything is working fine, except that I'm getting errors while trying to aknowledge or clear events. After adding all devices and organizing them I started to clear all the events that happen normally when first adding devices, but when I select one, two, some or all the events in the different devices and I click acknowledge or clear, it doesn't do anything for about one minute and popup a message saying "There was an error handling your request" and also a yellow bar appears at the top for a second saying "ZepConnectionError: Timed out connecting to service".

 

I have looked at all the logs and can't find something conclusive. Please guide me what could this mean and where to look for logs or hits about this error.

 

Thanks in advance!

 

printscreen of the messages is attached.

Attachments:
  • bobrog Newbie 1 posts since
    Sep 24, 2012
    Currently Being Moderated
    1. Sep 24, 2012 3:45 PM (in response to Pavel Perez)
    Re: zepConnectionError timeout service

    I am seeing a similar issue on 4.2.0 Core.  Sometimes when I try to drill into an event the details do not appear and "ZepConnectionError: Timed out connecting to service" is logged in event.log.

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009
    Currently Being Moderated
    2. Dec 5, 2012 3:55 PM (in response to bobrog)
    Re: zepConnectionError timeout service

    I'm seeing the same issue as the poster on one of my instances.  I've tried a ton of different stuff (including completely removing, reinstalling, and reconfiguring rabbitmq, and I still have the same issue.  I sure would like to know a fix for this in case I run in to it again in the future but because of deadlines I may have to just give up and re-build this box.  I'm working on troubleshooting this for the next little while.  I'll post back here if I find a solution.

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009
    Currently Being Moderated
    3. Dec 5, 2012 3:56 PM (in response to Pavel Perez)
    Re: zepConnectionError timeout service

    I'll add that this happened right out of the blue.  All I did was add 250 network devices without disabling monitoring of the interfaces and left it sitting there.  Since my instance polls for ifOperStatus and generates alerts it was sitting with over 5000 events in the console for a few days until I got back to working on it.  It was then that I noticed the problem.

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    4. Dec 6, 2012 11:11 AM (in response to Ryan Matte)
    Re: zepConnectionError timeout service

    Ryan:

     

    I want to resume trying to get this fixed later today, ping me in IRC.

     

    --Shane

  • Scott Burdick Newbie 1 posts since
    Jan 7, 2013
    Currently Being Moderated
    5. Jan 7, 2013 3:14 PM (in response to Pavel Perez)
    Re: zepConnectionError timeout service

    Just ran into this problem today.  Any resolution?

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009
    Currently Being Moderated
    6. Jan 7, 2013 3:18 PM (in response to Scott Burdick)
    Re: zepConnectionError timeout service

    I tracked it down to a component that appeared to be in the system but not linked to anything.  Because of deadlines I had to rebuild the whole box but I have the screwed up image saved.  I'll take another crack at fixing it when I have some time.  Apparently the bug that causes this sort of thing to occur is fixed in the next version, but that's no good for people that it has happened to.

  • Ryan Matte ZenossMaster 653 posts since
    Mar 26, 2009
    Currently Being Moderated
    7. Jan 7, 2013 3:19 PM (in response to Scott Burdick)
    Re: zepConnectionError timeout service

    Here is the Jira ticket that I opened for the issue: http://jira.zenoss.com/jira/browse/ZEN-4565

  • Rob Eagle Rank: Green Belt 155 posts since
    Feb 20, 2012
    Currently Being Moderated
    8. Jun 21, 2013 9:42 AM (in response to Ryan Matte)
    Re: zepConnectionError timeout service

    Looks like ticket might show resolution for 4.2.3, but am now experiencing this exact same issue after upgrading from 4.2.0 to 4.2.3 about a month ago (issue just started).

    Any ideas how to resolve?

    --Rob

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    9. Jun 24, 2013 5:28 AM (in response to Rob Eagle)
    Re: zepConnectionError timeout service

    Rob,

     

    Have you tried clearing your zep indexs?

     

    --Shane

  • Rob Eagle Rank: Green Belt 155 posts since
    Feb 20, 2012
    Currently Being Moderated
    10. Jun 24, 2013 6:47 AM (in response to Shane Scott)
    Re: zepConnectionError timeout service

    Shane,

    First off thanks for the reply, but no, only a reindex().  Not sure how that task would be completed?  Can you point me to a reference?

    --Rob

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    11. Jun 24, 2013 6:56 AM (in response to Rob Eagle)
    Re: zepConnectionError timeout service

    Rob,

     

    Ssh to the master then su zenoss; zopectl stop; zeneventserver stop; rm -Rf $ZENHOME/var/zeneventserver/index; zeneventserver start; zopectl start

     

    --Shane Scott (Hackman238)

    http://linkedin.com/in/shanewilliamscott

  • Rob Eagle Rank: Green Belt 155 posts since
    Feb 20, 2012
    Currently Being Moderated
    12. Jun 24, 2013 8:39 AM (in response to Shane Scott)
    Re: zepConnectionError timeout service

    Shane,

    commands ran fine and system came up but still same issue, any ideas?

    --Rob

     

    From event.log:

    2013-06-24T07:37:17 INFO extdirect Direct request failed: Timed out connecting to service.: EventsRouter.acknowledge [{'evids': [], 'params': {'severity': [5, 4, 3], 'eventClass': '/status/ping*', 'eventState': [0, 1, 2], 'tags': []}, 'limit': 100, 'excludeIds': {}}]

    ------

    2013-06-24T07:37:17 INFO extdirect DirectRouter suppressed the following exception (Response b29b2948-a82a-4809-8593-a6c7d286e370):

    Traceback (most recent call last):

      File "/opt/zenoss/Products/ZenUtils/extdirect/router.py", line 179, in _processDirectRequest

        response.result = _targetfn(**data)

      File "<string>", line 2, in acknowledge

      File "/opt/zenoss/Products/Zuul/decorators.py", line 69, in wrapped_fn

        return f(self, *args, **kwargs)

      File "/opt/zenoss/Products/Zuul/routers/zep.py", line 537, in acknowledge

        limit=limit,

      File "/opt/zenoss/Products/Zuul/facades/zepfacade.py", line 410, in acknowledgeEventSummaries

        limit)

      File "/opt/zenoss/lib/python/zenoss/protocols/services/zep.py", line 162, in acknowledgeEventSummaries

        limit=limit)

      File "/opt/zenoss/lib/python/zenoss/protocols/services/zep.py", line 102, in updateEventSummaries

        search_uuid = self.createSavedSearch(event_filter=event_filter, exclusion_filter=exclusion_filter, timeout=timeout)

      File "/opt/zenoss/lib/python/zenoss/protocols/services/zep.py", line 191, in createSavedSearch

        status, response = self.client.post(url, query)

      File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 175, in post

        return self._request_with_body(path, method='POST', body=body, params=params, headers=headers)

      File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 170, in _request_with_body

        response, content = self._executeRequest(request)

      File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 142, in _executeRequest

        raise self._connection_error_class('Timed out connecting to service.', e)

    ZepConnectionError: Timed out connecting to service.

  • Shane Scott ZenossMaster 1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    13. Jun 24, 2013 8:41 AM (in response to Rob Eagle)
    Re: zepConnectionError timeout service

    Rob,

     

    Can you post your zeneventserver.log and mysql log?

     

    --Shane

  • Rob Eagle Rank: Green Belt 155 posts since
    Feb 20, 2012
    Currently Being Moderated
    14. Jun 24, 2013 8:49 AM (in response to Shane Scott)
    Re: zepConnectionError timeout service

    Shane,

    Can't seem to find how to attach the logs to the post - Going to try replying to the email?

    --Rob

1 2 3 Previous Next

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points