Archived community.zenoss.org | full text search
Skip navigation
Up to Discussions in zenoss-users
2149 Views 8 Replies Latest reply: Sep 11, 2013 2:16 PM by kenhen93 RSS
kenhen93 Rank: White Belt
kenhen93
100 posts since
Sep 27, 2011
Currently Being Moderated

Aug 27, 2013 10:10 AM

Zenhub zopectl won't start...rabbitmq connection_closed_abruptly

Hi,

 

I am having a problem with Zenoss 4.2.4 running on Red Hat Enterprise Linux Server release 6.4    2.6.32-358.el6.x86_64. I have searched the zenoss community and internet pretty extensively.

 

When I start zenoss zenhub and zope will not start. Everything points to rabbitmq but I am not getting any good information from the logging of rabbitmq.

 

/var/log/rabbitmq/rabbit@localhost.log

=WARNING REPORT==== 27-Aug-2013::09:04:23 ===

closing AMQP connection <0.278.0> (127.0.0.1:36093 -> 127.0.0.1:5672):

connection_closed_abruptly

 

This is the message I get whenever I try to start zenhub.

 

Here are some other logs I have been looking at:

 

# rabbitmqctl status

Status of node rabbit@localhost ...

[{pid,47180},

{running_applications,[{rabbit,"RabbitMQ","2.8.7"},

                        {os_mon,"CPO  CXC 138 46","2.2.7"},

                        {sasl,"SASL  CXC 138 11","2.1.10"},

                        {mnesia,"MNESIA  CXC 138 12","4.5"},

                        {stdlib,"ERTS  CXC 138 10","1.17.5"},

                        {kernel,"ERTS  CXC 138 10","2.14.5"}]},

{os,{unix,linux}},

{erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:24:24] [rq:24] [async-threads:30] [kernel-poll:true]\n"},

{memory,[{total,31029512},

          {processes,11371088},

          {processes_used,11366544},

          {system,19658424},

          {atom,1139905},

          {atom_used,1138863},

          {binary,1053232},

          {code,11368014},

          {ets,2881632}]},

{vm_memory_high_watermark,0.4},

{vm_memory_limit,60839985152},

{disk_free_limit,1000000000},

{disk_free,1029694262272},

{file_descriptors,[{total_limit,924},

                    {total_used,18},

                    {sockets_limit,829},

                    {sockets_used,8}]},

{processes,[{limit,1048576},{used,236}]},

{run_queue,0},

{uptime,1845}]

...done.

 

/etc/rabbitmq/rabbitmq-env.conf

RABBITMQ_NODENAME=rabbit@localhost

RABBITMQ_MNESIA_BASE=/opt/zenoss/rabbitmq

RABBITMQ_MNESIA_DIR=/opt/zenoss/rabbitmq/mnesia/rabbit\@localhost

 

#rabbitmqctl list_vhosts

Listing vhosts ...

/zenoss

/

...done.

 

# rabbitmqctl list_users

Listing users ...

guest   [administrator]

zenoss  []

...done.

 

# rabbitmqctl list_queues -p /zenoss

Listing queues ...

celery  0

zenoss.queues.zep.modelchange   0

zenoss.queues.zep.signal        0

zenoss.queues.zep.migrated.summary      0

zenoss.queues.zep.rawevents     0

zenoss.queues.zep.heartbeats    0

m-prd-zen2.advertising.aol.com.celeryd.pidbox   0

zenoss.queues.zep.zenevents     0

zenoss.queues.zep.migrated.archive      0

...done.

 

# rabbitmqctl list_user_permissions zenoss

Listing permissions for user "zenoss" ...

/zenoss .*      .*      .*

...done.

 

/opt/zenoss/log/zenhub.log

2013-08-26 15:44:18,225 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,224 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused

2013-08-26 15:44:18,225 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,224 CRITICAL zen.Events: Unable to publish event to <Products.ZenMessaging.queuemessaging.publisher.EventPublisher object at 0x5380c50>: Could not connect to RabbitMQ: [111] Connection refused

2013-08-26 15:44:18,250 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,250 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused

2013-08-26 15:44:18,251 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,250 CRITICAL zen.Events: Unable to publish event to <Products.ZenMessaging.queuemessaging.publisher.EventPublisher object at 0x5380c50>: Could not connect to RabbitMQ: [111] Connection refused

2013-08-26 15:44:18,271 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,270 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused

2013-08-26 15:44:18,271 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,271 CRITICAL zen.Events: Unable to publish event to <Products.ZenMessaging.queuemessaging.publisher.EventPublisher object at 0x5380c50>: Could not connect to RabbitMQ: [111] Connection refused

 

/opt/zenoss/log/event.log

2013-08-27T09:05:40 ERROR ZODB.Connection Couldn't load state for 0x01d101

Traceback (most recent call last):

  File "/opt/zenoss/lib/python/ZODB/Connection.py", line 860, in setstate

    self._setstate(obj)

  File "/opt/zenoss/lib/python/ZODB/Connection.py", line 914, in _setstate

    self._reader.setGhostState(obj, p)

  File "/opt/zenoss/lib/python/ZODB/serialize.py", line 613, in setGhostState

    obj.__setstate__(state)

TypeError: __setstate__() takes exactly 2 arguments (1 given)

 

/opt/zenoss/log/zenactiond.log

2013-08-27 09:04:49,695 INFO zen.zenoss.protocols.amqp: RabbitMQ connection was closed: [Errno 104] Connection reset by peer

 

/opt/zenoss/etc/global.conf

amqphost localhost

amqpport 5672

amqpvhost /zenoss

amqpuser zenoss

amqppassword zenoss

amqpusessl 0

amqpadminport 55672

amqpadminusessl 0

 

This did work initially and i don't believe anything has changed so i don't know why it is not working all of the sudden. Thanks for any help!

 

Ken

Tags: 4.2
  • jcurry ZenossMaster
    jcurry
    1,021 posts since
    Apr 15, 2008
    Currently Being Moderated
    1. Sep 3, 2013 5:37 AM (in response to kenhen93)
    Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly

    Could this be my usual bete noir of disks filling up?

    Cheers,

    Jane

  • kenhen93 Rank: White Belt
    kenhen93
    100 posts since
    Sep 27, 2011
    Currently Being Moderated
    2. Sep 4, 2013 8:56 AM (in response to jcurry)
    Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly

    Hi Jane,

     

    Thanks for the reply. I wish it were that easy of an answer but disks look fine.

  • kenhen93 Rank: White Belt
    kenhen93
    100 posts since
    Sep 27, 2011
    Currently Being Moderated
    3. Sep 4, 2013 4:07 PM (in response to kenhen93)
    Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly

    Can anyone explain how zenhub uses rabbitmq? How many connections does it initiate? Does it open and close them?

     


    On a working version of zenoss I see in the rabbitmq log

     

    =INFO REPORT==== 4-Sep-2013::16:00:01 ===

    accepting AMQP connection <0.615.0> (127.0.0.1:50411 -> 127.0.0.1:5672)

     

    =INFO REPORT==== 4-Sep-2013::16:00:01 ===

    closing AMQP connection <0.615.0> (127.0.0.1:50411 -> 127.0.0.1:5672)

     

    on my non-working version I see

     

    =INFO REPORT==== 4-Sep-2013::15:49:48 ===

    accepting AMQP connection <0.294.0> (127.0.0.1:43124 -> 127.0.0.1:5672)

     

    =WARNING REPORT==== 4-Sep-2013::15:49:48 ===

    closing AMQP connection <0.294.0> (127.0.0.1:43124 -> 127.0.0.1:5672):

    connection_closed_abruptly

  • jcurry ZenossMaster
    jcurry
    1,021 posts since
    Apr 15, 2008
    Currently Being Moderated
    4. Sep 5, 2013 5:42 AM (in response to kenhen93)
    Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly

    I have just seen the same thing on a 4.2.4 system.  All requests for rabbitmqctl info got a response like:

     

    [root@zen42 rabbitmq]# rabbitmqctl -p /zenoss list_queues

    Listing queues ...

    Error: unable to connect to node rabbit@zen42: nodedown

     

    DIAGNOSTICS

    ===========

     

    nodes in question: [rabbit@zen42]

     

    hosts, their running nodes and ports:

    - zen42: [{rabbitmqctl5921,51984}]

     

    current node details:

    - node name: rabbitmqctl5921@zen42

    - home dir: /var/lib/rabbitmq

    - cookie hash: WxE1AhHRrbuaQLykIGgbaQ==

     

    Tried stopping rabbit and it said it wasn't running!

     

     

    [root@zen42 rabbitmq]# /etc/init.d/rabbitmq-server stop

    Stopping rabbitmq-server: RabbitMQ is not running

    rabbitmq-server.

    [root@zen42 rabbitmq]# ps -ef | grep rabbit

    rabbitmq  2194     1  0 09:57 ?        00:00:00 /usr/lib64/erlang/erts-5.8.5/bin/epmd -daemon

    root      6469  4889  1 10:22 pts/0    00:00:00 grep rabbit

     

    Manually killed this rabbitmq process and restarted rabbit with:

     

    service rabbitmq-server start

    Starting rabbitmq-server: SUCCESS

    rabbitmq-server.

     

    ps -ef shows LOTS more rabbit processes.

    rabbitmqctl commands work.

    zenhub.log is now clean of messages to do with rabbit.

    Everything seems OK.

     

    Strange thing was that zenoss status was reporting all daemons as running (whereas I think you said zenhub and zopectl wouldn't start?).

     

    It looks like rabbit had only half come up and forcing a complete restart got things going again.

     

    I would be very interested to know if this also cures the problem in your environment.

     

    Cheers,

    Jane

     

     

    BTW - I don't have your /etc/rabbitmq/rabbitmq-env.conf . Is this something you created??

  • kenhen93 Rank: White Belt
    kenhen93
    100 posts since
    Sep 27, 2011
    Currently Being Moderated
    5. Sep 5, 2013 10:00 AM (in response to jcurry)
    Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly

    Hi Jane,

     

    Yes I have noticed how when stopping rabbitmq '/usr/lib64/erlang/erts-5.8.5/bin/epmd -daemon' it still running. If I kill that process and start rabbitmq back up that process starts running again. I don't understand why Rabbitmq init script would just leave that process running. I tried this and it did not help with what is going on.

     

    Yes I created rabbitmq-env.conf so that I could force the mnesia location and hostname. I wanted the mnesia location to be located on a SAN drive that is mounted on my server and force the hostname to be localhost instead of the actual hostname.

  • jcurry ZenossMaster
    jcurry
    1,021 posts since
    Apr 15, 2008
    Currently Being Moderated
    6. Sep 5, 2013 10:17 AM (in response to kenhen93)
    Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly

    "and force the hostname to be localhost instead of the actual hostname"....

     

    Does this mean that the hostname has potentially changes as far as rabbit is concerned??

     

    Have you seen this append - message/71139#71139 ?

     

    It still sounds like rabbit is actively refusing to respond to zenhub so it has to be some sort of authentication mis-match.

     

    Do you now have lots of processes if you do ps -ef / grep rabbit ???

     

    Cheers,

    Jane

  • kenhen93 Rank: White Belt
    kenhen93
    100 posts since
    Sep 27, 2011
    Currently Being Moderated
    7. Sep 5, 2013 10:57 AM (in response to jcurry)
    Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly

    ok let me explain my setup better

     

    I am actually running zenoss on a 2 node veritas cluster. The zenoss filesystem, MySQL DB, rabbitmq mensia is all on a SAN that can fail back and forth between both nodes. This is why I use the rabbitmq-env.conf to make the mnesia location on the SAN and also force the hostname to be localhost (as both nodes are localhost).

     

    None of the Apps (Zenoss, MySQL, RabbitMQ) have any idea that this is a cluster.

     

    Zenoss works great on Node 1 but when I failover to node 2 I get the problem I reported in the first post where zenhub cannot connect to rabbitmq. This worked originally on the original failover test but will not now. I cannot find a reason why. Everything is the same on both nodes. The erlang cookie shouldn't matter because rabbitmq in not in a cluster mode. The erland cookie is also on on the SAN so it is passed back and forth when rabbitmq is failover.

     

    I didn't really want to get into this as I don't want this to get too confusing.

     

    On a failover to node 2, rabbitmq starts

     

    ps -ef | grep rabbitmq

    rabbitmq 28171     1  0 Sep04 ?        00:00:00 /usr/lib64/erlang/erts-5.8.5/bin/epmd -daemon

    root     41777     1  0 10:50 ?        00:00:00 runuser rabbitmq --session-command /usr/sbin/rabbitmq-server

    rabbitmq 41781 41777  0 10:50 ?        00:00:00 /bin/sh /usr/sbin/rabbitmq-server

    rabbitmq 41791 41781  3 10:50 ?        00:00:02 /usr/lib64/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /opt/zenoss/rabbitmq -- -noshell -noinput -sname rabbit@localhost -boot /opt/zenoss/rabbitmq/rabbit@localhost-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@localhost.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@localhost-sasl.log"} -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/opt/zenoss/rabbitmq/mnesia/rabbit@localhost"

    rabbitmq 41971 41791  0 10:50 ?        00:00:00 inet_gethost 4

    rabbitmq 41972 41971  0 10:50 ?        00:00:00 inet_gethost 4

    root     45458 40092  0 10:51 pts/0    00:00:00 grep rabbitmq

     

    rabbitmqctl list_connections

    Listing connections ...

    zenoss  127.0.0.1       43314   running

    zenoss  127.0.0.1       43288   running

    zenoss  127.0.0.1       43301   running

    zenoss  127.0.0.1       43284   running

    zenoss  127.0.0.1       43277   running

    zenoss  127.0.0.1       43313   running

    zenoss  127.0.0.1       43297   running

    ...done.

     

    service zenoss status

    Daemon: zeneventserver program running; pid=43636

    Daemon: zopectl not running

    Daemon: zenrrdcached program running; pid=43835

    Daemon: zenhub not running

    Daemon: zenjobs program running; pid=45012

    Daemon: zeneventd program running; pid=45063

    Daemon: zenping program running; pid=45153

    Daemon: zensyslog program running; pid=45205

    Daemon: zenstatus program running; pid=45242

    Daemon: zenactiond program running; pid=45280

    Daemon: zentrap program running; pid=45315

    Daemon: zenmodeler program running; pid=45368

    Daemon: zenrender program running; pid=45398

    Daemon: zenperfsnmp program running; pid=45438

    Daemon: zencommand program running; pid=45487

    Daemon: zenprocess program running; pid=45517

    Daemon: zenmail program running; pid=45558

    Daemon: zredis program running; pid=45561

    Daemon: zenjmx program running; pid=45604

    Daemon: zenwinperf program running; pid=45650

    Daemon: zenwin program running; pid=45693

    Daemon: zeneventlog program running; pid=45746

     

    zenhub start

    (/opt/zenoss/log/zenhub.log)

    2013-09-04 15:48:27,231 INFO zen.ZenHub: Worker (28552) reports 2013-09-04 15:48:27,230 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused

    2013-09-04 15:48:27,231 INFO zen.ZenHub: Worker (28552) reports 2013-09-04 15:48:27,231 CRITICAL zen.Events: Unable to publish event to <Products.ZenMessaging.queuemessaging.publisher.EventPublisher object at 0x5eb77d0>: Could not connect to RabbitMQ: [111] Connection refused

    2013-09-04 15:48:27,246 INFO zen.ZenHub: Worker (28552) reports 2013-09-04 15:48:27,245 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused

    2013-09-04 15:48:27,246 INFO zen.ZenHub: Worker (28552) reports 2013-09-04 15:48:27,245 CRITICAL zen.Events: Unable to publish event to <Products.ZenMessaging.queuemessaging.publisher.EventPublisher object at 0x5eb77d0>: Could not connect to RabbitMQ: [111] Connection refused

    2013-09-04 15:48:27,267 INFO zen.ZenHub: Worker (28552) reports 2013-09-04 15:48:27,267 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused

     

    /var/log/rabbitmq/rabbitmq@localhost.log

    =INFO REPORT==== 5-Sep-2013::10:53:08 ===

    accepting AMQP connection <0.350.0> (127.0.0.1:43405 -> 127.0.0.1:5672)

     

    =WARNING REPORT==== 5-Sep-2013::10:53:09 ===

    closing AMQP connection <0.350.0> (127.0.0.1:43405 -> 127.0.0.1:5672):

    connection_closed_abruptly

     

    free

                 total       used       free     shared    buffers     cached

    Mem:     148535120    4199564  144335556          0     349992    1838108

    -/+ buffers/cache:    2011464  146523656

    Swap:      4194288          0    4194288

     

    df

    Filesystem           1K-blocks      Used Available Use% Mounted on

    /dev/mapper/vg0-root   1032088    377348    602312  39% /

    tmpfs                 74267560         0  74267560   0% /dev/shm

    /dev/sde1               148742     33391    107671  24% /boot

    /dev/mapper/vg0-home   1032088     34176    945484   4% /home

    /dev/mapper/vg0-opt    8256952    845196   6992328  11% /opt

    /dev/mapper/vg0-srv  558356472    202408 530014544   1% /srv

    /dev/mapper/vg0-usr    4128448   1602120   2316616  41% /usr

    /dev/mapper/vg0-var    4128448    855076   3063660  22% /var

    none                   1048576    238772    809804  23% /tmp

    tmpfs                        4         0         4   0% /dev/vx

    /dev/vx/dsk/zenmasterDG/zenoss-vol01

                         1073708032   1369014 1005317891   1% /opt/zenoss

  • kenhen93 Rank: White Belt
    kenhen93
    100 posts since
    Sep 27, 2011
    Currently Being Moderated
    8. Sep 11, 2013 2:16 PM (in response to kenhen93)
    Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly

    This was fixed by a reboot. I hate to say that

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points