Configuration : How to install distributed collectors : Comments

RE: How to install distributed collectors

community@zenoss.org — Fri, 11 Dec 2009 21:07:03 GMT

I am creating a new data collector and removed zenhub from $ZENHOME/etc/daemons.txt.

Should the following Zenoss daemons run on a data collector? If so, how is they used?

zeoctl
zopectl
zenjobs

- Ken

RE: How to install distributed collectors

community@zenoss.org — Sat, 12 Dec 2009 00:16:46 GMT

I am sorry if the question sounds trivial but I have not set up a data collector before.

I am able to connect to the data collector using a standard URL http://zenoss.noida.in:8080. Should the Render URL : http://zenoss.paris.fr:8090/zenoss.noida.in to a data collector render in a browser or is this used internally by Zenoss under the covers to render RRD files from the data collector?

When I attempt to connect to a data collector which is up and running using the 8090 path listed above, I get a page with the following error. Please let me know

web.Server Traceback (most recent call last):

exceptions.IndexError: list index out of range

RE: How to install distributed collectors

community@zenoss.org — Mon, 14 Dec 2009 19:21:50 GMT

@jenkinskj Using the instructions above, accessing the collector at it's IP at 8080 should not work. What did you do to enable that ability? And can you see performance data?

I added zeoctl and zopectl to the daemons list above and it is listening at 8080 but not responding.

My understanding is that the performance data stays on the collector.

RE: How to install distributed collectors

community@zenoss.org — Mon, 14 Dec 2009 19:43:11 GMT

For the collector to work via URL via port 8080, Zope and ZOE was enabled on the collector. I suspect that these two daemons should not have been set up to run on the collector.

I made an attempt to remove zeoctl and zopectl from the daemons.txt file, stopped Zenoss and restarted it and ZEO and ZOPE daemons are still running on the collector. I suspect that we need to manally remove the daemons from startring up on the collector.

You are correct RRD performance data stays on the collector. What I wanted to find out about is if the collector render URL should display anything via port 8090 or is this URL used internally to pass RRD data from collector to main server. When I try to connect to the URL http://zenoss.paris.fr:8090/zenoss.noida.in via a browser, I get a bunch of errors.

- Ken

RE: How to install distributed collectors

community@zenoss.org — Mon, 14 Dec 2009 19:57:01 GMT

I think when you removed zopectl and zeoctl from daemons.txt, they were skipped on shutdown. If you manually stop those and restart the stack, they will not come back up.

Can you see rrd graphs of monitored devices in the collector's interface?

I see the same errors when going to that render URL above.

RE: How to install distributed collectors

community@zenoss.org — Mon, 14 Dec 2009 20:20:19 GMT

Updating the daemons.txt file as you suggested on the collector worked. Thanks for sharing.

As far as the rrd graphs go on the collector's interface, I have not created any devices just yet.So, I am not able to see the graphs until I move a device over to the collector.

I need to set customize the RRD create commands before monitoring devices on the collector and on the main server. For this test server, I would like to contain one month of data. Do you have any examples for RRD AVERAGE and MAX that I can replace the settings default with? For our Production environment, I will need 1 year of data.

- Ken

RE: How to install distributed collectors

community@zenoss.org — Mon, 14 Dec 2009 21:45:48 GMT

Based on errors in the log file, I do not think that we should be running Zenjobs daemon on a data collector (i.e. if ZEO does not run on the collector).

2009-12-14 14:30:16,564 WARNING ZEO.zrpc: (21141) CW: error connecting to ('localhost', 8100): ECONNREFUSED

Can someone confirm this?

- Ken

RE: How to install distributed collectors

community@zenoss.org — Wed, 13 Jan 2010 20:09:04 GMT

After some research, it looks like ZEO and ZOPE need to be running as daemons. Per the recent edits above zope.conf is updated and pointing back to the main Zenoss server. Should I configure ZEO to reference the main Zenoss server as well or should this be running on the collector?

- Ken

RE: How to install distributed collectors

community@zenoss.org — Thu, 14 Jan 2010 00:05:15 GMT

I have this going with only the services listed above. I don't think zeo and zope need to run on the collector. Plus, I think zope pointing back to the main server is only for remote hubs.

RE: How to install distributed collectors

community@zenoss.org — Thu, 14 Jan 2010 00:14:28 GMT

The reason I thought ZEO and ZOPE needed to be started was because zenactions reported the following WARNINGs.

2009-12-14 14:28:32,651 WARNING ZEO.zrpc: (21276) CW: error connecting to ('loca
lhost', 8100): ECONNREFUSED

Once I started ZEO and ZOPE on the collector, the WARNING went away.

Is there anything special that has to be done to set up zenactions on the collector?

Also I did not start ...

zensyslog
zenactions
zentrap

Should these be running on the collector? 

Are they working for you?

- Ken

RE: How to install distributed collectors

community@zenoss.org — Wed, 17 Feb 2010 21:10:04 GMT

zentrap and zensyslog should be running on the collector, zenactions should not, I've edited this document to reflect that.

RE: How to install distributed collectors

community@zenoss.org — Thu, 01 Apr 2010 21:34:46 GMT

I have a remote collector I'd like to add in on a NATted network - unfortunately it's also my first attempt at a distributed collector so I think things are a little more complicated that on a wide-open network.

I'm having two problems: 1, zenmodeler can't find any plugins, and 2, I have no idea which network port(s) I have to punch through the firewall to this distributed collector to make it work seamlessly.

Any suggestions on the firewall ports would be very helpful.

The zenmodeler log looks like this (I love all the lines stating "No module named ."):

2010-03-30 13:10:22,068 INFO zen.ZenModeler: Run as a daemon, waiting 22 seconds to start.
2010-03-30 13:10:22,069 INFO zen.ZenModeler: Connecting to --thezenosshubserver--:8789
2010-03-30 13:10:22,709 INFO zen.ZenModeler: Connected to ZenHub
2010-03-30 13:10:45,847 INFO zen.ZenModeler: Collecting for path /Devices
2010-03-30 13:10:46,493 ERROR zen.ZenModeler: Problem loading plugin zenoss.snmp.NewDeviceMap
2010-03-30 13:10:46,494 ERROR zen.ZenModeler: Traceback (most recent call last):
File "/bigdisk/zenoss/Products/DataCollector/Plugins.py", line 102, in create
self.modPath)
File "/bigdisk/zenoss/Products/DataCollector/Plugins.py", line 149, in importPlugin
plugin_pkg = imp.find_module('.', [package])
ImportError: No module named .

2010-03-30 13:10:46,494 ERROR zen.ZenModeler: Due to import errors, removing the zenoss.snmp.NewDeviceMap plugin from this collection cycle.
2010-03-30 13:10:46,494 ERROR zen.ZenModeler: Problem loading plugin zenoss.snmp.DeviceMap
2010-03-30 13:10:46,494 ERROR zen.ZenModeler: Traceback (most recent call last):
File "/bigdisk/zenoss/Products/DataCollector/Plugins.py", line 102, in create
self.modPath)
File "/bigdisk/zenoss/Products/DataCollector/Plugins.py", line 149, in importPlugin
plugin_pkg = imp.find_module('.', [package])
ImportError: No module named .

2010-03-30 13:10:46,494 ERROR zen.ZenModeler: Due to import errors, removing the zenoss.snmp.DeviceMap plugin from this collection cycle.
2010-03-30 13:10:46,495 ERROR zen.ZenModeler: Problem loading plugin zenoss.snmp.InterfaceMap
2010-03-30 13:10:46,495 ERROR zen.ZenModeler: Traceback (most recent call last):
File "/bigdisk/zenoss/Products/DataCollector/Plugins.py", line 102, in create
self.modPath)
File "/bigdisk/zenoss/Products/DataCollector/Plugins.py", line 149, in importPlugin
plugin_pkg = imp.find_module('.', [package])
ImportError: No module named .

etc... through all the plugins it tries to load.

RE: How to install distributed collectors

community@zenoss.org — Thu, 13 May 2010 22:43:10 GMT

It appears that some presumptions made by others here are incorrect.

I have followed the directions *exactly* as specified in this document in two separate setups and in both instances no jobs would ever make it to the remote collector.

That is, until zeoctl and zopectl were enabled on the remote system.

Now that I've added those in to the list of daemons to start, zeo was able to create some storage that it apparently needed, and there was a lot of communication to the master server on port 8100 (as expected after reading through the conf files).

The key thing for me was why this documentation suggests configuring the remote system to communicate with the remote zope on port 8100 but there was nothing set on the master to listen on 8100, nor was there ever any attempt by the remote collector to connect to the master on 8100.

First step - one has to change zeo.conf on the master server to not listen only on localhost:8100. The config line needs to read "address :8100", then restart zenoss.

Second - add zeoctl and zopectl (I have no proof currently if zopectl is necessary, but I suspect it is, considering the conf for zope to connect to the master zope) to the distributed collector.

Voila, it fired up, started doing something with the devices it was supposed to start collecting for, and everything is showing up on the master, including graphs, as it's supposed to.

==========================

Further to this - after a successful launch with zeoctl and zopectl on the remote collector, things worked, but every time I add a device on the master, nothing happens at the remote collector. If I shut down and restart zenoss on the remote collector, it picks up on the new devices it needs to work with and models them and starts polling them.

I've tried removing zeoctl and zopectl on the remote collector, and the behaviour is exactly the same. The remote collector doesn't pick up on changes from the master until I restart zenoss.

Are we sure there's not something else that's supposed to run on the remote collector?

RE: How to install distributed collectors

community@zenoss.org — Fri, 14 May 2010 15:16:57 GMT

I've followed these steps and had success with it without having zeoctl and zopectl running on the collector, so I'm not sure where you're going wrong. There is one small detail that isn't mentioned here. When you're specifying monitor and hubhost in each of the collector's config files, monitor can not be 127.0.0.1. It has to be the IP of a network interface on the server. If you leave it as 127.0.0.1 nothing is going to work.

Edit: I added it to the steps.

RE: How to install distributed collectors

community@zenoss.org — Fri, 14 May 2010 15:48:45 GMT

If you're having difficulty with these steps I'd recommend that you try one of the following ZenPacks:

http://community.zenoss.org/docs/DOC-5831
http://community.zenoss.org/docs/DOC-5861

RE: How to install distributed collectors

community@zenoss.org — Thu, 08 Jul 2010 15:07:12 GMT

Have you seen where the graphs work in the perf tab under a specific device, but if I create a multigraph report, the graphs do not work. THey worked fine before adding the second collector.

RE: How to install distributed collectors

community@zenoss.org — Thu, 12 Aug 2010 19:30:14 GMT

I think this document need a new revision. It´s don´t work on Zenoss 3.0.1. I´m working no an new version. But I´m having some problems with zenrender on the remote collector.

RE: How to install distributed collectors

community@zenoss.org — Fri, 13 Aug 2010 16:34:28 GMT

Agreed. I have been working on this for a while not and haven't been able to get the distributed collector up and running.

This http://community.zenoss.org/docs/DOC-5861 references the page we are at now. It says that the zenpack should work on 3.0. I just haven't been able to get it working yet.

RE: How to install distributed collectors

community@zenoss.org — Fri, 13 Aug 2010 17:06:00 GMT

Then I would suggest posting what issue you're running in to on that ZenPack page so that Egor can have a look. I'll have to update my Collector Tool pack as well once we've figured out what the problems are.

RE: How to install distributed collectors

community@zenoss.org — Fri, 13 Aug 2010 17:11:40 GMT

try change 'Render URL' of remote collector to: http://dnsnameofremotecollector:8091

RE: How to install distributed collectors

community@zenoss.org — Fri, 13 Aug 2010 17:34:34 GMT

Still having problems with multigraph reports

2010-08-13 12:30:12,963 WARNING RenderServer: /tmp/renderserver/graph-e6b20af71912e0a32237bfc40648aa8d129600.png --start=1281590712 --end=1281720312 COMMENT:2010-08-12 0\:30\:12\t\t\t to \t\t\t2010-08-13 12\:30\:12\c --imgformat=PNG -F -E --height=100 --vertical-label= COMMENT:MISSING RRD FILE\: <SERVERNAME> AvailableKBytes COMMENT:MISSING RRD FILE\: <SERVERNAME> AvailableKBytes --width=500 --start=end-129600s --end=now-0s
2010-08-13 12:30:12,968 DEBUG zen.zenrender: Processing render request
2010-08-13 12:30:12,969 ERROR RenderServer: Failed to generate a graph
Traceback (most recent call last):
File "/usr/local/zenoss/zenoss/Products/ZenRRD/RenderServer.py", line 190, in render
rrdtool.graph(*gopts)
error
2010-08-13 12:30:12,969 WARNING RenderServer: /tmp/renderserver/graph-4631e427b3ed558d817f4a554d847e44129600.png --start=1281590712 --end=1281720312 COMMENT:2010-08-12 0\:30\:12\t\t\t to \t\t\t2010-08-13 12\:30\:12\c --imgformat=PNG -F -E --height=100 --vertical-label= COMMENT:MISSING RRD FILE\: <SERVERNAME> AvailableKBytes COMMENT:MISSING RRD FILE\: <SERVERNAME> AvailableKBytes --width=500 --start=end-129600s --end=now-0s
2010-08-13 12:30:12,974 DEBUG zen.zenrender: Processing render request
2010-08-13 12:30:12,975 ERROR RenderServer: Failed to generate a graph
Traceback (most recent call last):
File "/usr/local/zenoss/zenoss/Products/ZenRRD/RenderServer.py", line 190, in render
rrdtool.graph(*gopts)
error

RE: How to install distributed collectors

community@zenoss.org — Fri, 13 Aug 2010 20:20:08 GMT

Thanks that worked great. Do you have any issues with the zoom not working. When I click on the zoom it tried to download the image instead of zooming.

RE: How to install distributed collectors

community@zenoss.org — Wed, 25 Aug 2010 19:26:29 GMT

Which are the ports I need to open in firewall?

RE: How to install distributed collectors

community@zenoss.org — Wed, 25 Aug 2010 19:59:12 GMT

from what it looks like here you need port 8091 open if you are using a distributed collector

RE: How to install distributed collectors

community@zenoss.org — Wed, 01 Sep 2010 14:35:45 GMT

All,

I am using the zencollectortool zenpack and all seems good but I am unable to zoom in on graphs that were collected on a remote collector. Might anyone have insight into this?

Thanks,

RE: How to install distributed collectors

community@zenoss.org — Thu, 02 Sep 2010 20:20:07 GMT

I just wanted to verify that when you do all these steps, is it suppose to make it so you can no longer connect to the collector interface via http://collector01:8080

RE: How to install distributed collectors

community@zenoss.org — Fri, 03 Sep 2010 17:47:54 GMT

Question on 5.1: RRD performance data are to be copied to the remote server

This is the command I used:

scp -Cr $ZENHOME/perf/Devices/mydevice.co.company collector01.co.company:$ZENHOME/perf/Devices

Is this step intended to dump all RRD templates into the $ZENHOME/perf/Devices? I would think this would be a problem
Should there not be a prior step to the scp command?

zenoss@collector01:mkdir $ZENHOME/perf/Devices/mydevice.co.company

If you make the directory for the device then the command will be changed to

scp -Cr $ZENHOME/perf/Devices/mydevice.co.company collector01.co.company:$ZENHOME/perf/Devices/mydevice.co.company

RE: How to install distributed collectors

community@zenoss.org — Fri, 03 Sep 2010 18:24:02 GMT

Allan, Egor's ZenPack will do all that for you, there's no need for you to do it by hand.

RE: How to install distributed collectors

community@zenoss.org — Fri, 03 Sep 2010 18:47:53 GMT

When does it do that?

When you change the collector for the device from localhost to collector01.co.company?

RE: How to install distributed collectors

community@zenoss.org — Fri, 03 Sep 2010 18:51:16 GMT

When you move a device from the localhost collector to the remote collector, or vice-versa.

RE: How to install distributed collectors

community@zenoss.org — Fri, 03 Sep 2010 18:56:54 GMT

Other then SSH command line, is there a way to interface with the remote collector since http://collector01:8080 does not work anymore?

Sorry, just trying to understand this. Does all monitoring happen through the master http://master:8080, but is stored and collected through the collector?

RE: How to install distributed collectors

community@zenoss.org — Fri, 03 Sep 2010 18:59:18 GMT

I think you're missing the point. The remote collector just runs the collection daemons which all feed the data back to the main Zenoss instance. Everything is done on the main Zenoss instance (except for the collection, obviously).

RE: How to install distributed collectors

community@zenoss.org — Fri, 03 Sep 2010 19:05:02 GMT

Ok, I tend to complicate easy.

Is there an ability to have a device collecting from a remote collector and the main Zenoss instance?

RE: How to install distributed collectors

community@zenoss.org — Fri, 03 Sep 2010 19:09:28 GMT

No, it's one or the other.

RE: How to install distributed collectors

community@zenoss.org — Fri, 03 Sep 2010 19:13:30 GMT

Ok, thanks for your patience with all of these questions.

RE: How to install distributed collectors

community@zenoss.org — Fri, 03 Sep 2010 19:15:03 GMT

No problem, for future reference, the Zenoss IRC channel on irc.freenode.net:6667 - #zenoss is quite active. It's a good first place to try for support on issues such as this.

RE: How to install distributed collectors

community@zenoss.org — Mon, 06 Sep 2010 18:56:28 GMT

Hi, Finally I have finished my fresh Zenoss Core 3 install with remote Mysql server and a Remote collector.

But I can't get it working. My graphs don't work.

I have open a tread to it:

http://community.zenoss.org/thread/14422

RE: How to install distributed collectors

community@zenoss.org — Thu, 21 Jul 2011 11:09:47 GMT

I've just got two remote collectors working, one on the same network as the master server and one in a DMZ.

I believe that our networks people have opened up the correct firewall ports to allow the DMZ collector and master servers to talk to each other but it appears that the master server is running all the SNMP modelling of a device in the DMZ, not the collector.

Is this what should be happening?

I didn't notice this on the collector server that's on the same network as the master since the server I decided to monitor with that collector was accessible to both machines via SNMP.

I was under the impression that all checks; WMI, SNMP etc would be run from the collector?