ZenPacks : Distributed Collectors : Comments

RE: Distributed Collectors

community@zenoss.org — Wed, 28 Apr 2010 19:39:37 GMT

This looks like it will be a very useful ZenPack to have.

If I already built distributed collectors by hand (essentially created a script to stop Zenoss, tar and scp the ZenPacks and Products directory from the primary server to collectors), can I replace my script with this ZenPack to keep the collectors updated moving forward?

Does this ZenPack manage the creation of required links for ZenPacks which require daemons to be added. I had to build those by hand (i.e. zenperfwmi, zenperfodbc)?

Please let me know.

RE: Distributed Collectors

community@zenoss.org — Wed, 28 Apr 2010 21:21:09 GMT

Hi jenkinskj,

1. you can try.

2. yes, it does.

Egor

RE: Distributed Collectors

community@zenoss.org — Wed, 28 Apr 2010 21:22:30 GMT

Thank you!

RE: Distributed Collectors

community@zenoss.org — Sun, 02 May 2010 21:11:15 GMT

Hiya,

Did you try this with manually set up collectors?

Regards

RE: Distributed Collectors

community@zenoss.org — Mon, 03 May 2010 13:37:18 GMT

Not yet. I plan to but have to finish up with testing a few more ZenPacks first.

RE: Distributed Collectors

community@zenoss.org — Tue, 04 May 2010 11:23:48 GMT

I went ahead and installed this but noticed it missed a rudimentary step, as was pointed out in a previous post... zeo.conf needs to be modified to not bind to localhost only!

#26.4.1. Prerequisite Tasks

By default, only local access to the ZEO database is configured. Before adding a remote hub, you must edit the$ZENHOME/etc/zeo.conf file to allow remote access.

In the file, change the line:

address localhost:8100

address 8100

RE: Distributed Collectors

community@zenoss.org — Tue, 04 May 2010 12:02:34 GMT

Also, the following daemons were not started on the collector, nor were their configuration files set up correctly - therefore leading to unrenderable graphs:

zenmodeler
zenrender

RE: Distributed Collectors

community@zenoss.org — Tue, 04 May 2010 17:11:49 GMT

Hello Dayton Turner,

1. ZenPacks.zenoss.DistributedCollector != ZenPacks.community.DistributedCollectors

2. After you install ZenPacks.community.DistributedCollectors you need to restart zenoss.

Egor

RE: Distributed Collectors

community@zenoss.org — Tue, 04 May 2010 17:15:13 GMT

Egor,

1. Yep I know, I installed the zenpack attached here, but then had to begin looking when it wouldnt quite work

2. Pretty sure i did that, a few times. It didnt work until i modified the daemons.txt on the remote collector and enabled zenrender and zenmodeler. Have i done it wrong?

Dayton

RE: Distributed Collectors

community@zenoss.org — Tue, 04 May 2010 18:07:37 GMT

Dayton,

by every Remote Monitors Update, file daemons.txt for remote collector will be generated from list of daemons running on Master server except ['zeoctl','zopectl','zenhub','zenjobs','zenactions','zenmodeler'] daemons. Check if zenrender daemon running on Master server. I'm not sure, if we need zenmodeler on remote collector.

Egor

RE: Distributed Collectors

community@zenoss.org — Tue, 04 May 2010 18:46:23 GMT

Egor,

Yes, zenrender is infact running on the Master server. I presume its normal, but there is no daemons.txt on the master server.

I wonder why zenrender was not included in the daemons.txt generated on the remote monitor?

Dayton

RE: Distributed Collectors

community@zenoss.org — Tue, 04 May 2010 20:12:12 GMT

Yeh, zenrender should be running on both the master and the collector, so that'll need to be fixed.

RE: Distributed Collectors

community@zenoss.org — Tue, 04 May 2010 20:38:48 GMT

I have installed this ZenPack. Received no ERRORS but when I click on the Add Remote Monitor it just hangs on loading. I have installed this on "Solaris 5.10 Generic_141445-09 i86pc i386 i86pc"

Also Egor, I have Zenoos Master on a Solaris Zone, and my first collector is on the same GLOBAL server so I am running it with a different username. In your plugin can I change the username I will SSH with? Is it hard coded or using zProperties?

RE: Distributed Collectors

community@zenoss.org — Wed, 05 May 2010 00:57:15 GMT

I also notice that when i try to zoom on a graph that was collected on the remote collector , it never zooms. Why might this be?

RE: Distributed Collectors

community@zenoss.org — Wed, 05 May 2010 01:16:51 GMT

Interesting. My main Zenoss node has zenrender running as a process, but Settings -> Daemons shows zenrender is not started. Clicking start changes nothing.

RE: Distributed Collectors

community@zenoss.org — Tue, 11 May 2010 21:04:20 GMT

Hi Horsemann,

sorry, i have no experience with Solaris In DC ZenPack SSH used without any username, so it connect to remote collector as 'zenoss' user.

Egor

RE: Distributed Collectors

community@zenoss.org — Fri, 14 May 2010 15:51:25 GMT

Did you restart Zenoss after enabling zenrender?

RE: Distributed Collectors

community@zenoss.org — Fri, 14 May 2010 15:51:56 GMT

I'm not sure about the zooming, I didn't have that issue with my distributed collectors.

RE: Distributed Collectors

community@zenoss.org — Fri, 14 May 2010 19:29:39 GMT

On two different zenoss hubs (one installed as a zenpack, the other from source), both throw event log errors when installing this module (version 1.1) reading:

2010-05-14T14:38:21 ERROR Zope.SiteErrorLog http://sysmon:8080/zport/dmd/ZenPackManager/manage_installZenPack
Traceback (innermost last):
Module ZPublisher.Publish, line 119, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 42, in call_object
Module Products.ZenModel.ZenPackManager, line 294, in manage_installZenPack
Module ZServer.HTTPResponse, line 163, in write
TypeError: Value must be a string

The zenpack still shows as installed, but all of the Add/Update/Delete Remote Monitor(s) menu options just brings up a "Loading..." dialog box in the browser (tested over firefox & google chrome) with a spinner and it never changes to anything else.

Both zenoss setups are running 2.5.2

RE: Distributed Collectors

community@zenoss.org — Fri, 14 May 2010 19:32:21 GMT

Did you restart Zenoss after installing the ZenPack? You get this error during installation?

RE: Distributed Collectors

community@zenoss.org — Fri, 14 May 2010 20:14:31 GMT

Yes to both.

RE: Distributed Collectors

community@zenoss.org — Thu, 05 Aug 2010 21:55:21 GMT

Zenoss 3.0

I installed SSH on the remote machine that will be the remote collector with "ssh public key authentication.

When I try to connect to that machine using ssh zenoss@ip address I get the error below.

ssh zenoss@#### -i .ssh/id_dsa

ssh: /usr/local/zenoss/common/lib/libcrypto.so.0.9.8: no version information available (required by ssh)

Segmentation fault

I believe that this is why my graphs are not pulling into the main zenoss collector. I am getting the error

Throughput

There was a problem rendering this graph. Either the file does not exist or an error has occurred. Initial graph creation can take up to 5 minutes. If the graph still does not appear, look in the Zope log file $ZENHOME/log/event.log for errors.

on all my graphs.

RE: Distributed Collectors

community@zenoss.org — Thu, 05 Aug 2010 22:06:12 GMT

Hi derek654,

check if this work for you on zenoss machine:

su zenoss
ssh remotecollectorsip

RE: Distributed Collectors

community@zenoss.org — Fri, 06 Aug 2010 15:31:46 GMT

changed 'Render URL' of remote collector to: http://dnsnameofremotecollector:8091.

Fixed.

Thanks

RE: Distributed Collectors

community@zenoss.org — Fri, 13 Aug 2010 20:19:16 GMT

I am having issues with the zooming not working. I was wondering if you found what is causing this? When I click to zoom it tries to download the image.

it says

You have chosen to open

render

Which is a: image/html

from http://10.10.101.82:8091 which is what I have set as the render url.

RE: Distributed Collectors

community@zenoss.org — Sat, 21 Aug 2010 04:05:48 GMT

I am having same issue on zomming, is there any idea to solve this problem?

RE: Distributed Collectors

community@zenoss.org — Wed, 01 Sep 2010 17:41:28 GMT

http://community.zenoss.org/docs/DOC-5861

You are not the only one with this problem.

RE: Distributed Collectors

community@zenoss.org — Wed, 01 Sep 2010 17:45:52 GMT

Yeh, no one appears to have found a solution yet but we're going to have to start trying.

RE: Distributed Collectors

community@zenoss.org — Thu, 02 Sep 2010 20:18:05 GMT

I can run the ssh command manually at a command line and it will prompt me for a user ID. So the question is where and how do you configure the user ID and password for the remote collector. I have the SSH set for the zProperties on the device and collectors Render User and Password.

This is the error I get when running the Update Remote Collector portion of this Zenpack. I am getting the following error:

Remote Collector collector01.us.company

Stopping zenoss daemons

Permission denied, please try again.

Permission denied (publickey,password).

Revert Remote Collector configuration

Permission denied, please try again.

Permission denied (publickey,password).

Remove ZenPacks files from Remote Collector

Permission denied, please try again.

Permission denied (publickey,password).

Copy ZenPacks files to Remote Collector

Permission denied, please try again.

Permission denied (publickey,password).

cpio: write error

: Broken pipe

find: `standard output'

: Broken pipe

find: write error

Update Remote Collector configuration

Permission denied, please try again.

Permission denied (publickey,password).

Starting zenoss daemons

Permission denied, please try again.

Permission denied (publickey,password).

Finish

RE: Distributed Collectors

community@zenoss.org — Thu, 02 Sep 2010 20:28:58 GMT

When he says "configure public key ssh authentication" in the install steps he means this: http://community.spiceworks.com/education/projects/Passwordless_SSH_Using_Shared_Keys

You don't configure the username or password anywhere in Zenoss. You setup a shared key so that you can login as the zenoss user to the remote collector passwordless.

RE: Distributed Collectors

community@zenoss.org — Fri, 03 Sep 2010 14:30:44 GMT

Thanks, Ryan. That of course worked. I felt the 'doh - smacked to head moment'

Now I am getting the following error when running the Update Monitor:

Command Output

Remote Collector collector01.co.company

Stopping zenoss daemons

bash: zenoss: command not found

Revert Remote Collector configuration

Update zenping configuration

/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.community.DistributedCollectors-1.1-py2.4.egg/ZenPacks/community/DistributedCollectors/bin/updateConfigs: 47: cannot create /etc/zenping.conf: Permission denied

find: `/ZenPacks'

: No such file or directory

When I sign on to the collector01.co.company it is able to run the zenoss command, so I am not sure why this is showing an error. I am able to ssh from master to collector, run the bash command and then the zenoss command with no errors
Where is the update looking for /etc/zen*.conf . I have verified that the .conf files are in /usr/local/zenoss/zenoss/etc
Also where is it looking for /Zenpacks, on collector01.co.company and the master it is in /usr/local/zenoss/zenoss/ZenPacks

RE: Distributed Collectors

community@zenoss.org — Fri, 03 Sep 2010 15:26:26 GMT

That is very obviously an environment issue when it's logging in. For whatever reason no Zenoss environment variables are being picked up.

I ran in to this same problem a while back. The solution is to create a file on the remote collector in /home/zenoss/.ssh called environment. The end path to the file will be /home/zenoss/.ssh/environment.

Populate the file with the following lines:

PATH=/usr/local/zenoss/common/sbin:/usr/local/zenoss/python/bin:/usr/local/zenoss/mysql/bin:/usr/local/zenoss/zenoss/bin:/usr/local/zenoss/common/bin:/sbin:/usr/sbin:/usr/local/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
ZENHOME=/usr/local/zenoss/zenoss
PYTHONHOME=/usr/local/zenoss/python
DYLD_LIBRARY_PATH=/usr/local/zenoss/python/lib:/usr/local/zenoss/mysql/lib:/usr/local/zenoss/zenoss/lib:/usr/local/zenoss/common/lib::
PYTHONPATH=/usr/local/zenoss/python/lib/python2.4/site-packages:/usr/local/zenoss/common/lib:/usr/local/zenoss/zenoss/lib/python:/usr/local/zenoss/python/lib:
ZENOSS_LD_LIBRARY_PATH=/usr/local/zenoss/python/lib:/usr/local/zenoss/mysql/lib:/usr/local/zenoss/zenoss/lib:/usr/local/zenoss/common/lib::

Then save the file and try again.

RE: Distributed Collectors

community@zenoss.org — Fri, 03 Sep 2010 16:03:13 GMT

I have set this environment variable, but I am still getting the same errors.

I ssh to collector01.co.company and verified all the paths. They are all showing correctly but DYLD_LIBRARY_PATH is coming up with blank variables. Unsure why at this point other then a variable setting that may be trumping or removing it.

RE: Distributed Collectors

community@zenoss.org — Fri, 03 Sep 2010 16:18:18 GMT

Executing a remote command directly over SSH (like the ZenPack is doing) and actually logging in to a shell via ssh are 2 completely different things. To test it properly you'd have to do something like:

ssh -l username hostname 'cat $ZENHOME/etc/zenperfsnmp.conf'

RE: Distributed Collectors

community@zenoss.org — Fri, 03 Sep 2010 16:23:26 GMT

Sorry, I just remember something else that you need to do. Edit your /etc/ssh/sshd_config file and add the following line to the end of it:

PermitUserEnvironment yes

Then restart sshd:

/etc/init.d/ssh restart

RE: Distributed Collectors

community@zenoss.org — Fri, 03 Sep 2010 17:02:27 GMT

Ryan, thank you.

I am now able to run a clean "Update Monitor" with no errors

Adding the environment variables and setting the PermitUserEnvironment in the sshd_config was the key to completing the configure "ssh public key authentication"

It would be nice if the information from http://community.spiceworks.com/education/projects/Passwordless_SSH_Using_Shared_Keys

and the creation of the environment file and modification of the sshd_config could be added to these instructions.

Regards,

Allan

RE: Distributed Collectors

community@zenoss.org — Fri, 03 Sep 2010 17:12:56 GMT

You're welcome, I'll work with egor on getting a more specific guide written up and added to this page.

RE: Distributed Collectors

community@zenoss.org — Mon, 13 Sep 2010 07:07:06 GMT

hi, I have a question.

I have a device being monitor by collector which run apache and mysql. Now I need 300 secs(default) to collect memory&disk info and 60s to collect interface counter(ethernetCsmacd_64).

How to edit? I only find "SNMP Performance Cycle Interval" in collector->edit, but it change all config.

How to config the check period of other performance tools such as apache&mysql??

thanks

RE: Distributed Collectors

community@zenoss.org — Mon, 13 Sep 2010 16:46:52 GMT

Zenoss does not support individual collection cycles for different snmp values. Collection cycles can only be set for the actual daemons. What you are asking is not currently possible.

RE: Distributed Collectors

community@zenoss.org — Mon, 20 Sep 2010 22:06:53 GMT

OOooooooook. The zoom problem was driving me nuts, so I had to track it down.

I know WHY it's broken, but Javascript hurts my head so bad that I don't have a proposed fix:

The zoom in/out is controlled by a bunch of javascript in the file zenrrdzoom.js, on your master Zenoss server (located at /opt/zenoss/Products/ZenModel/skins/zenmodel/zenrrdzoom.js for me). To load the new graph images, the script uses some trickery that changes document.location.hash to include the graph ID of the image that it should be updating:

registerResponse: function() {
var response = document.location.hash;

This ONLY works if the graph renderer is the same host as your Zenoss master; javascript same-origin code prevents the remote collector URLs from being able to modify the document.location variable. Until FireFox 3.6.2, you could get away with using window.location.hash to do the same thing, but that's been patched now, as well.

I'd be curious to see how/if Zenoss Enterprise handles this. Or maybe zooming of remote graphs doesn't work in Enterprise, either?

RE: Distributed Collectors

community@zenoss.org — Mon, 20 Sep 2010 22:11:28 GMT

OH, yeah, there's a second bug in the remote render:

The "save as" bug that comes up when you click a graph, and it says you have a file of type "image/html..."

This is due to a change in the way python's mimetypes module handles "guess_type." In the remote renderer process (zenrender.py), there is this line:

mimetype = mimetypes.guess_type('%s' % ftype)[0]

This line fails, as guess_type now expects a full filename. I changed mine to look like this:

mimetype = mimetypes.guess_type('*.%s' % ftype)[0]

The Save As problem went away, as it's now sending the correct content-type (text/html). The zooming is still broken, of course, due to the javascript issue I mentioned above.

RE: Distributed Collectors

community@zenoss.org — Mon, 20 Sep 2010 22:35:18 GMT

This just got fixed in a recent Enterprise ticket verified against 3.0.2, http://dev.zenoss.org/trac/log/trunk/Products/ZenModel/skins/zenmodel/zenrrdzoom.js is the source and #2527 was the internal ticket.

RE: Distributed Collectors

community@zenoss.org — Tue, 21 Sep 2010 13:13:43 GMT

Are you saying that the zooming issue was fixed as well?

RE: Distributed Collectors

community@zenoss.org — Tue, 21 Sep 2010 13:59:06 GMT

I'm saying there was an Enterprise ticket closed with the description "cannot zoom on a graph from a remote collector" and the only fixes went into zenrrdzoom.js. I don't know the specifics of how the Enterprise or Community Distributed Collectors work, but that this may have been fixed and someone using the Community version should make sure they have the latest version of the file.

RE: Distributed Collectors

community@zenoss.org — Tue, 21 Sep 2010 14:12:39 GMT

Cool, was just looking for some clarification which you've provided. He had also mentioned an issue with right clicking and saving the file. I thought you might have been talking about that. Thank you.

RE: Distributed Collectors

community@zenoss.org — Thu, 14 Oct 2010 03:00:47 GMT

Hi Ryan, I've been having lots of issues getting a remote collector to talk with my hub. I've tried not to hassle anyone here but I think it would be great if there was a good description somewhere of how these 2 machines communicate. Then I could fix the issues myself.

For instance- we know that SSH public key encryption is mandatory, but what ports are needed? I have been advised to use 8091, but when testing I find that my collector does not respond to this port, but it does to 8789.....

I'm off to upgrade Zenoss to 3.0.2 and distributed collectors v1.6. We'll see what happens then.

I am happy to contribute by writing up any solutions I find, but I need to understand what's going on first!

Adam

RE: Distributed Collectors

community@zenoss.org — Thu, 14 Oct 2010 14:05:48 GMT

I honestly don't know all of the ports involved. I don't use distributed collectors in production, I just dabbled with them when I was writing my collector tool zenpack which is obsolete now that Egor released his Distributed Collectors pack. Why don't you work with tcpdump/netstat and see what ports it's trying to go out through?

RE: Distributed Collectors

community@zenoss.org — Fri, 15 Oct 2010 01:21:44 GMT

Thanks for the reply, and suggestions. I'll keep going and see how far I can get.

cheers

Adam

RE: Distributed Collectors

community@zenoss.org — Tue, 04 Jan 2011 15:44:13 GMT

Is the graph Zooming issue fixed already?

RE: Distributed Collectors

community@zenoss.org — Wed, 05 Jan 2011 01:21:57 GMT

Yes it fix on Zenoss 3.0.3 for me.

Kelvin Lo

RE: Distributed Collectors

community@zenoss.org — Mon, 10 Jan 2011 12:44:09 GMT

Hi everyone. I didn't find the case to send the answer correctly but i beleive its better have the information even in the wrong place.

Regarding empty or not displayed graphs when using zenoss distributed collectors,the solution is install the Master server with the same name inside on *.conf files on collectors as one nromal device(normally appers Localhost) and change , the zenrender.conf or directly via Web to daemons - zenrender - view config on Master server ,to Monitor Localhost and Hubhost Localhost.

I had the same problem and i solved change those two things.

I hope it'll help!

Best Regards.

RE: Distributed Collectors

community@zenoss.org — Thu, 17 Feb 2011 18:33:34 GMT

Hi.. i have a great deal of problems installing the remote collectors.. :S

I use Ubuntu Server 10.10 64 bit on both the Remote collector and on the Hub..

My problem starts at the SSH keys.. i do the following:

1. Login in as Zenoss (By sudo su -l zenoss.... the reason for this is i dont know the password for the standard zenoss user in stack installation :S)

2. use ssh-keygen -t.... and run the the steps without passwords etc.

It's here my problem starts.. when it creates the keys it does not save the public key as it should, it saves it as ?.pub and not in the folder .ssh/ (i have testet with my regular user there it works fine, i simply dont know whats wrong)?

I cant copy the key over either because i dont know the zenoss users password...

Thanks..

RE: Distributed Collectors

community@zenoss.org — Fri, 18 Feb 2011 09:46:56 GMT

I tried this morning on ubuntu 10.04 instead.. and got the same result, i dont get it?..

It's only the Zenoss user there is the problem, on every other user i can make keys, but on the zenoss user it simply don’t make the identities, it saves the .pub as a different sign and then .pub and not in the right folder..

zenoss@RemoteCollector03:~$ ssh-keygen -t rsa

ssh-keygen: /usr/local/zenoss/common/lib/libcrypto.so.0.9.8: no version information available (required by ssh-keygen)

Generating public/private rsa key pair.

Enter file in which to save the key (/home/zenoss/.ssh/id_rsa):

Created directory '/home/zenoss/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in B.

Your public key has been saved in B.pub.

The key fingerprint is:

c3:c7:8d:fe:46:26:1b:bc:d8:e7:41:bb:6a:03:c2:bb zenoss@m

The key's randomart image is:

+--[ RSA 2048]----+

| |

| . . o |

| . S.+ o |

| o .++.o. |

| o +.Bo |

| . . *.oo |

| E. ..*+ |

+-----------------+

zenoss@RemoteCollector03:~$ ls

B.pub

zenoss@RemoteCollector03:~$

Can somebody help with this??

Thanks

RE: Distributed Collectors

community@zenoss.org — Thu, 24 Feb 2011 12:38:58 GMT

The problem was solved by the changing the OS to CentOS... Weird but worked!

Another question:

Is auto discovery a working feature for remote collectors?

(I fill the Default Discovery Network setting in the Collector but nothing happens, and when i run an auto discovery on the specific scope Zenoss contacts the LocalHub's collector to do this, anyway to change collector in the auto discovery?)

RE: Distributed Collectors

community@zenoss.org — Thu, 24 Feb 2011 16:05:28 GMT

Distributed Collectors are not officially supported in Core. This is essentially a community hack to get them to function. There is no way to specify what collector to perform an auto discovery with in Core. Auto discoveries are very messy anyways, you'd be better off adding stuff in by hand.

RE: Distributed Collectors

community@zenoss.org — Thu, 24 Feb 2011 18:09:00 GMT

Hey Ryan...

Thank you for your reply, sounds like the oldschool way is the way to go..

RE: Distributed Collectors

community@zenoss.org — Tue, 01 Mar 2011 10:50:51 GMT

Hey...

I have a simple question..

Can you change the SSH port (i do not want the Hub to send a SSH request on port 22, to the RemoteCollector)

If this is possible.. how to?

Thanks in advance...

RE: Distributed Collectors

community@zenoss.org — Tue, 01 Mar 2011 22:50:53 GMT

Hi Lumskekonig,

for now SSH calls are hardcoded in ZenPack. so you have 3 options:

1. modify ZenPacks __init__.py file and replace all 'ssh' calls with 'ssh -p yourportnumber'

2. add 'Port' parameter to /home/zenoss/.ssh/config file

3. add 'Port' parameter to /etc/ssh/ssh_config file

Egor

RE: Distributed Collectors

community@zenoss.org — Thu, 03 Mar 2011 18:42:24 GMT

Hi Egor

Used the third.. Worked like a charm

Thank u..

RE: Distributed Collectors

community@zenoss.org — Tue, 08 Mar 2011 08:23:54 GMT

Hi...

One more question...

Did anyone ever solve the graph problem... i can zoom and navigate on the "Hub" but not on any devices on the remote collector... Weird?..

Any insight on this subject?

RE: Distributed Collectors

community@zenoss.org — Tue, 08 Mar 2011 19:43:02 GMT

Actually, there was a fix for this done in Enterprise that involved replacing a certain javascript file. Matt Ray had pointed it out to me. I guess the fix didn't make it's way in to core. I'm busy at the moment but I'll dig up the file when I have time.

RE: Distributed Collectors

community@zenoss.org — Tue, 08 Mar 2011 20:45:59 GMT

Hey Ryan...

Could it be this one http://dev.zenoss.org/trac/browser/trunk/Products/ZenModel/skins/zenmodel/zenrrdzoom.js?rev=25389

In all circumstances where to replace the file? The hub or the Remote Collector?....

Thank you for your quick answer..

Peter Brøndum

RE: Distributed Collectors

community@zenoss.org — Wed, 09 Mar 2011 09:15:03 GMT

I've just checked the file i linked to and the file in my zenoss installation... They match.

So i hope you have another file for me..

Best Regards

Peter Brøndum

RE: Distributed Collectors

community@zenoss.org — Wed, 09 Mar 2011 15:57:45 GMT

I found the following:

http://dev.zenoss.org/trac/ticket/7453

They claim it was fixed, but evidently not if it's still not working for you. I think the patch was only part of it, the replacement of a javascript file was the second part, but I forget what file it was. I'll ask to ask Matt Ray if he remember next time I see him.

RE: Distributed Collectors

community@zenoss.org — Wed, 30 Mar 2011 10:30:58 GMT

Hey Ryan...

Have you looked into the problem with Matt Ray... I dont understand if the patch was made in version 3.0.3 and use 3.1.0... my version should be updated..? hmmm..

Thanks in advance

RE: Distributed Collectors

community@zenoss.org — Wed, 30 Mar 2011 13:12:45 GMT

I asked him and he doesn't remember. The javascript file was part of an internal Zenoss ticket, so I don't have access to find it. I'll see if maybe Nick Yeates can dig it up.

RE: Distributed Collectors

community@zenoss.org — Wed, 30 Mar 2011 13:18:02 GMT

Oh bonkers...

well thanks a lot for your help... i'll se if any of the next updates has a fix..

Best regards

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 19:58:20 GMT

Hi Egor,

I was just wondering if you might explain why you chose not to do remote zenhubs and rather have the remote collectors talk back to a central zenhub. Looking at your ZenPack code I don't think it would be that hard for me to modify it to fit my needs but before going that route was wondering if there was a technical/performance reason you chose not to do that.

Thanks,

--Dennis

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 20:46:55 GMT

Hi Dennis,

Discrtibuted Collectors ZenPack implements this scenario. Feel free to extend it with remote zenhub support.

Egor

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 20:47:27 GMT

Dennis, that's not even possible with core to my knowledge. You have one zope, one zenhub, one mysql, multiple collector daemons. That's the architechture. I also don't see what performance benefit that would bring, since you can always increase the number of zenhub threads (workers) on the hub. Generally you want to keep collectors as light weight as possible.

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 21:02:41 GMT

Maybe I'm misunderstanding. But let's say zenhub on the central server dies (maybe the box reboots, or some other reason), no events could get into the mysql database. If a remote collector has a local zenhub then if the main zenhub dies events could still get into the database. Am I wrong?

--Dennis

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 21:13:37 GMT

I can speak to parts of this... In the new Zenoss 3.*, we updated many of the daemons to do store and fwd abilities on remote collectors. Meaning, they have some kind of local cache in case the master goes down. Because we focused these daemon updates to the workings of enterprise distributed architecture (we dont touch core distributed arch's), this may or may not work. Core distributed collectors are done via ZenPacks developed by community members - a totally different piece of code than the enterprise one that we spec'ed the store and forward to.

I am really not sure, but it is worth looking into if you have interest here.

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 21:15:00 GMT

dhopp perhaps with enough fiddling around you could get something like that working, but zenhub is very stable, so it's not really an issue unless you get up to tens of thousands of devices, at which point you'd want to get the enterprise version anyways just for the extra distributed features.

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 21:32:05 GMT

Believe me, if it were up to me I would go with the Enterprise edition. Unfortunately, it's not completely up to me and so I have to play with the deck of cards I was dealt. I'm hopeful that after showing what's possible I'll get a lot of "can we do this or that" and can push them into Enterprise. I've read a few posts that have remote zenhubs working (granted they are very much a "it worked for me" type of post) and so was exploring it.

So what happens in the event that zenhub machine reboots? Are the events lost forever? Let's say zenhub machine goes down for a reboot and for some reason it takes 15 minutes (maybe it needs to check the filesystem), during this time an event happens. When zenhub comes back up the issue that triggered that event is still happening, do I get immediate visibility to that event (even if the count is just 1 rather then 4 or whatever it should be)? What about the remote collectors ability to read/write RRD files?

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 21:34:05 GMT

You would lose the events while it's down, you wouldn't lose the performance data since the RRDs are stored on the collectors themselves.

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 21:35:53 GMT

As soon as zenhub recovered if the event was still happening we would see it right?

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 21:41:27 GMT

Correct.

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 21:41:53 GMT

Though you would miss errors that come in once such as traps, or syslogs.

RE: Distributed Collectors

community@zenoss.org — Mon, 16 May 2011 21:47:31 GMT

Thanks Ryan! Initially we won't be worrying about traps or syslogs so I'm not worried about them (at this moment).

RE: Distributed Collectors

community@zenoss.org — Mon, 27 Jun 2011 08:10:05 GMT

Hi!

After some attemps, it finally worked ok for us... but now we have a pair of unsolved questions:

- We added <host> as a remote collector, it worked ok, but now we removed it. The performance data about the <host> collector (cpu, zendaemons...) is still being polled from the main server, so there's an event every <snmp cycle> seconds for each perfdata, indicating an error associated to a <host> device that doesn't exist... how should we clean it up? there's not any findable object.

- If you remove a remote collector while some devices are managed by it, is there any way to stablish the collector as the localhost one for those "lost devices"? The only way we found is getting into the remote collector and setting perfserver to "localhost", but if you don't have the remote collector, you don't have this path :S

Thanks in advance.

Luizz

RE: Distributed Collectors

community@zenoss.org — Mon, 27 Jun 2011 15:29:15 GMT

Hi Luizz,

1. remote colector object and Device object in Zenoss is two different things. So search for this Device and remove it from Zenoss.

2. hmm, what can i say...? Move devices to another collector BEFORE deletion.

Egor

RE: Distributed Collectors

community@zenoss.org — Tue, 28 Jun 2011 10:38:32 GMT

Hi bigegor,

the <host> i added as a remote collector is not a device, it doesn't appear in devices. It appears in the "Device" column on the event console... I promise, i never added it as a Device!!!

in the event console, it appears a line like the next for every component zentrap, zensyslog, zenstatus, zenrender...

1000,2011/06/28 12:32:35.000,unknown,,,,prbrpresidl05|zenwin|/Status/Heartbeat||4|prbrpresidl05 zenwin heartbeat failure,,,,2011/06/27 09:15:11.000,/Status/Heartbeat,...

referring to second question, i only needed to know if it were possible to bypass my "human failure behaviour" , it's ok.

Thanks for your fast response!

RE: Distributed Collectors

community@zenoss.org — Thu, 14 Jul 2011 12:17:07 GMT

Hi!

Finally i found the stuff with the heartbeat errors. Looking at the events DB in mysql, there's a table "heartbeat", which contains "device|component|timeout|lastTime" for checking, so it's not needed a host to be created as a device for beeing into the "to-check" list of heartbeat. Solution for my problem? Obviously, delete from heartbeat where device='hostname of the remote collector';

RE: Distributed Collectors

community@zenoss.org — Thu, 25 Aug 2011 21:53:17 GMT

Hi Egor,

I'm sure this is a user (me) error but I was hoping you could shed some light onto a problem I'm having. I started using this ZenPack recently and am loving it except 1 issue. When I try to update my remote collectors I get a Zenoss error:

Type:

Value: 'NoneType' object is not iterable

My assumption is I have a ZenPack installed that you're trying to copy to the remote collectors but for whatever reason it fails with this error.

Any suggestions on what I could do to troubleshoot? Besides removing the ZenPacks 1 by 1?

As always, thanks in advance.

Regards,

John

RE: Distributed Collectors

community@zenoss.org — Tue, 30 Aug 2011 14:17:34 GMT

As I suspected, user error. Would have been helpful to check the boxes next to the collectors I wanted to update.

RE: Distributed Collectors

community@zenoss.org — Fri, 09 Sep 2011 17:42:17 GMT

Hi There,

Has anyone tried this on Zenoss 3.2?

Those were the steps took:

1 - Installed and started Zenoss 3.2 (stack) on CentOS5.6 on the main server;

2 - Installed ZenPacks.community.DistributedCollectors-1.7.egg on the main server from UI(/zport/dmd/ZenPackManager) and restarted zenoss-stack;

3 - Installed and started Zenoss 3.2 (stack) on CentOS5.6 on the remote server as it was a stand alone server;

4 - Set up and tested the Public Key/No Password ssh login (main->remote and remote->main);

5 - From the master UI (/zport/dmd/Monitors) added the remote monitor by his domain name;

6 - Updated the Remote Monitor (got the yellow bar!) and restarted zenoss-stack on remote server;

7 - Disabled MySQL on the remote server (/usr/local/zenoss/mysql/scripts/ctl.sh stop);

8 - Added a device (linux server with snmpd running) with collector set to be the remote server.

And this is what I got:

Modeling the device worked.

Ping and snmpwalk commands worked as well.

Does not recognize changes (i.e. "eth1 down/up") by snmp polling or traps;

Shows device down only when I shutdown the manageIP (ping);

The performance graphs (viewDaemonPerformance or deviceDetailNav:IpInterface or deviceDetailNav:FileSystem) does not show any values ("nan");

Please share your experience with 3.2 and how you work around the problems.

Thanks in advance,

Felipe Castelar

RE: Distributed Collectors

community@zenoss.org — Mon, 26 Sep 2011 17:02:26 GMT

I've tried again with 3.02.

It does now recognize changes by snmp polling and traps. But I still can't see the perf. graphs.

Those two pics shows part of a pcap file I did with wireshark.

The first one shows the snmp response from 10.61.124.165 (monitored device) to 10.61.124.39 (remote collector -> "dsv-en-39"). See that it's receiveing data normally from the monitored device.

The second shows the zenhub comunication from 10.61.124.39 to 10.61.124.38 (my main server - the hub server - "dsv-en-38"). See that it's showing a "timeout message" "eventclass Heartbeat".

I can ping dsv-en-39 from dsv-en-38 (by name - /etc/hosts) and ssh with user zenoss without password.

The same I get pinging dsv-en-38 from dsv-en-39 (by name) and ssh with user zenoss without password.

Does any one have a clue what's going on?

Thanks in advance,

Felipe Castelar

RE: Distributed Collectors

community@zenoss.org — Wed, 04 Jan 2012 15:52:32 GMT

Hi Guys,

I am stuck in ssh common error. "Zenoss password"

[zenoss@ZENLAB01 ~]$ scp .ssh/id_rsa.pub zenoss@collector:.ssh/authorized_keys

zenoss@collector's password:

I am using centos5 and zenoss 3.0.3. I am sure you all would have crossed this one. Please help me with this. It is bit urgent as I have to get going with distributed collector as soon as possible.

Thanks & Regards

RE: Distributed Collectors

community@zenoss.org — Wed, 04 Jan 2012 17:14:36 GMT

This is a composition of a bunch of posts that I already made in this thread about setting up the ssh auth for this ZenPack so that it works properly...

When he says "configure public key ssh authentication" in the install steps he means this: http://community.spiceworks.com/education/projects/Passwordless_SSH_Using_Shared_Keys

After you've done that...

Create a file on the remote collector in /home/zenoss/.ssh called environment. The end path to the file will be /home/zenoss/.ssh/environment. As the zenoss user on the master type "env". Place the ouput in the file that you just created on the collector.

Edit /etc/ssh/sshd_config and add the following line to the end of the file: PermitUserEnvironment yes

Restart sshd.

RE: Distributed Collectors

community@zenoss.org — Mon, 09 Jan 2012 17:29:19 GMT

Hi Ryan,

Thanks for the reply. But still I have some issues in establishing the distributed collector setup. I did the same with bit change.

For example:

Main Server - Zen01

Remoter Collector - Zen02

Setting is as follows:

Main server, I installed zenoss 3.0.3 core on centos 5.0.5 manually stack installer.

and Remote collector, I am using virtual appliance.

I installed zenpack in main server (zen01).

I pushed the public key from main server (zen01) to root of the collector (Zen02) and in the collector, I moved the file to zenoss (/home/zenoss/.ssh) and change the ownership to zenoss. I did for bi-direction. I tested ssh commands between collector and main server on both side to confirm it works from command line. "ssh -l username hostname 'cat $ZENHOME/etc/zenperfsnmp.conf' " I enabled zenrender in both sides at /usr/local/zenoss/zenoss/bin/zenoss. I don’t find any file named Daemons_txt_only after installing zenpack and adding remonitor.

I added the remote monitor as zen02's ip address in Main server(zen01) through GUI without any error. I noticed SSH traffic are sent to and fro in the wireshark. So I assumed that it worked without adding environment file. (I hope it is fine.)

Before adding remote monitor (zen02), I have added a device to monitor in collector(zen02). I was under impression that after implementing collector idea, I should be seeing devices added, events and zenpacks from collector (Zen02) to device list, events and zenpacks in Main server (zen01). But, I don’t see any of them. I did update remote monitor. GUI shows that update has been done successfully and I see ssh traffic again to and fro in wireshark. But still I don’t see devices, events and zenpacks from collector(zen02)

I am not sure what is going wrong here. I see by default, its localhost for collector in both main server and collector. To troubleshoot, in collector (zen02) I changed localhost to collector's ip address in

"/usr/local/zenoss/zenoss/etc/zencommand.conf, zendisc.conf, zeneventlog.conf, zenmodeler.conf, zenperfsnmp.conf, zenping.conf, zenprocess.conf, zenrender.conf, zenstatus.conf, zensyslog.conf, zentrap.conf, zenwin.conf, zenwinmodeler.conf "

And added zen02's ip address as collector in Zen02 itself to check if the ip address is working fine to check with ip address setting is working locally before testing in main server (zen01). I can see the performance graph in the collector (zen02 side) but I can't see any performance graph in remote monitor across zen02's ip address in Main server zen01. Its all "nan" in Zen01 side.

Does it matters with main server's and collector's OS and all?? Please let me know is there anything else I have to change manually other than zenrender in the setup after installing zenpack and adding collector's ip address as remote monitor in main server and ssh setup.

Kindly help me with the same, I am literally breaking my head...

Thanks in advance.

RE: Distributed Collectors

community@zenoss.org — Mon, 09 Jan 2012 17:40:49 GMT

Don't mix different installers like that. If the master was installed using the stack installer then you should use the stack installer for the collector as well. The virtual appliance has zenoss in /opt/zenoss because it was installed with the RPM, while the stack installer puts it in /usr/local/zenoss. The environments between the two are setup differently. You're going to run in to all kinds of problems when you mix them like you've done. It's highly recommended to use the same OS and installer for all installations when doing a collector model.

RE: Distributed Collectors

community@zenoss.org — Mon, 09 Jan 2012 18:04:15 GMT

Big ouch... coz I am working on this for few days and breaking my head. But ya i was changing on required stuffs on /opt/zenoss in virtual installer side. Let me try with two virtual appliances setup so that I need not to install everything from scratch and I will update you.

But, other than that, everything else is correct. right?

One more question is do I need to change from localhost to servers' ip addresses in the confg file, as default it shows localhost in the collector section in both the servers? and I am using only ip addresses instead of domain name, it doesn't matter in logically but thought of confirming.

Thanks a lot Ryan.

RE: Distributed Collectors

community@zenoss.org — Mon, 09 Jan 2012 18:39:33 GMT

You need to edit the config files for the daemons on the collector and set "monitor" to the hostname of the collector (it can't be localhost or 127.0.0.1, it needs to be a hostname or IP that matches the interface to monitor from on the device). The "hubhost" line should be the hostname or IP of the master (not localhost or 127.0.0.1). I thought this ZenPack already handled all of that configuration though provided that SSH is setup correctly.

RE: Distributed Collectors

community@zenoss.org — Mon, 09 Jan 2012 19:26:10 GMT

I thought same about the zenpack that it will change the ip addresses in the config file. But, for me it didnt change. Both monitor and hubhost remained localhost.

Also, does Zenpack takes care of transfer devices added in collector to master from collector by itself or we have to do it manually?

5. TRANSFERT EXISTING DEVICES TO THE REMOTE DISTRIBUTOR

5.1 RRD performance data are to be copied to the remote server

zenoss.paris.fr# scp -Cr /usr/local/zenoss/zenoss/perf/Devices/mydevice zenoss.noida.in:/usr/local/zenoss/zenoss/perf/Devices

5.2 don't forget to add the remote collector in devices /etc/snmpd.conf

rocommunity public zenoss.noida.in

5.3 Move the devices to the remote collector

Management / Collectors / zenoss.paris.fr
Select the devices and use "Set collector"

Thanks for quick reply Ryan.

RE: Distributed Collectors

community@zenoss.org — Mon, 09 Jan 2012 19:36:02 GMT

As far as I remember Egor telling me it will transfer the RRDs over itself (I think it uses rsync). The RRDs are the only things that need to be transferred to the collector, everything else is stored in zope on the master. Since it didn't automatically configure the configuration files on the collector for you, it really sounds like you didn't setup ssh properly... try issuing a command as the zenoss user from the master to the collector via ssh like:

ssh zenoss@host "zenoss status"

You should get a direct return from the command without actually logging in to the server. If that doesn't work then it's not setup correctly.

RE: Distributed Collectors

community@zenoss.org — Mon, 16 Jan 2012 16:56:50 GMT

Hi Ryan,

Sorry for the delay reply... It is working... thanks for tht... main reason was zenoss installation as u mentioned earlier... one was virtual appliance and other one is stack installation... I believe the path used in the zenpack might be of stack installation one... and i manually enable zenrender. It is graphing...

I added the device in main server and specified collector as remote collector's ip address.Then i can see the device.

ssh was working even in previous setup too.. But, one question is I tried to login to other remote collector as http://1.1.1.2:8091 but i get line errors at some lines in python script? Do I need to change anything?

RE: Distributed Collectors

community@zenoss.org — Mon, 16 Jan 2012 18:52:52 GMT

Nah, the error is probably because you connected to the URL without passing it any arguments. I wouldn't worry about it.

RE: Distributed Collectors

community@zenoss.org — Sat, 28 Jan 2012 19:18:17 GMT

Felipe Castelar:

Know it is long ago, but have you tried to checking the zenrender daemon on both the LocalHub and the Collector.

I had succes by starting the Zenrender daemon seperatly and afterwards running a "zenoss restart" command.

RE: Distributed Collectors

community@zenoss.org — Sat, 28 Jan 2012 19:22:36 GMT

A quick question:

I have runned a system with remote collectors for some time now and have noticed that the collectors keep eating more and more of my memory, as if something is stored in the memory and never clears.

Has anyone expirenced this aswell? And is there a solution, other than restarting the machine?

I'm currently running on a CentOS 5,7 and zenoss 3.1 (Just updated to 3.2 the same happens).

Best Regards

RE: Distributed Collectors

community@zenoss.org — Wed, 08 Feb 2012 22:35:56 GMT

Just trying again?

Anyone else discovered this?, and any ideas?

Best Regards

RE: Distributed Collectors

community@zenoss.org — Tue, 21 Feb 2012 10:04:16 GMT

I have configured to get this worked but could not Model the devices from remote collector. Any device which is added as localhost collector could Model without problem. Any advise?

The error message I got:

2012-02-21 00:01:48,876 INFO zen.ZenModeler: Connecting to localhost.localdomain:8789

2012-02-21 00:02:18,915 ERROR zen.ZenModeler: Timeout connecting to zenhub: is it running?

Device with local collector could be model-ed:

2012-02-21 00:07:26,505 INFO zen.ZenModeler: Connecting to localhost:8789

2012-02-21 00:07:26,548 INFO zen.ZenModeler: Connected to ZenHub

2012-02-21 00:07:26,594 INFO zen.ZenModeler: Collecting for device 192.168.100.192

2012-02-21 00:07:26,690 INFO zen.ZenModeler: No WMI plugins found for 192.168.100.192

2012-02-21 00:07:26,693 INFO zen.ZenModeler: No Python plugins found for 192.168.100.192

2012-02-21 00:07:26,702 INFO zen.ZenModeler: No command plugins found for 192.168.100.192

2012-02-21 00:07:26,711 INFO zen.ZenModeler: SNMP collection device 192.168.100.192

2012-02-21 00:07:26,711 INFO zen.ZenModeler: plugins: zenoss.snmp.NewDeviceMap, zenoss.snmp.DeviceMap, zenoss.snmp.InterfaceMap, zenoss.snmp.RouteMap, zenoss.snmp.HRFileSystemMap, zenoss.snmp.UCDHardDiskMap

2012-02-21 00:07:26,780 INFO zen.ZenModeler: No portscan plugins found for 192.168.100.192

2012-02-21 00:07:28,613 INFO zen.SnmpClient: snmp client finished collection for 192.168.100.192

2012-02-21 00:07:28,614 INFO zen.ZenModeler: Processing zenoss.snmp.NewDeviceMap for device 192.168.100.192

2012-02-21 00:07:28,615 INFO zen.ZenModeler: processing zenoss.snmp.DeviceMap for device 192.168.100.192

2012-02-21 00:07:28,616 INFO zen.ZenModeler: Modeler zenoss.snmp.InterfaceMap processing data for device 192.168.100.192

2012-02-21 00:07:28,620 INFO zen.ZenModeler: processing zenoss.snmp.RouteMap for device 192.168.100.192

2012-02-21 00:07:28,623 INFO zen.ZenModeler: Modeler zenoss.snmp.HRFileSystemMap processing data for device 192.168.100.192

2012-02-21 00:07:28,624 INFO zen.ZenModeler: Skipping /proc/fs/nfsd. 0 total blocks.

2012-02-21 00:07:28,624 INFO zen.ZenModeler: Skipping /proc/sys/fs/binfmt_misc. 0 total blocks.

2012-02-21 00:07:28,624 INFO zen.ZenModeler: Skipping /var/lib/nfs/rpc_pipefs. 0 total blocks.

2012-02-21 00:07:28,632 INFO zen.ZenModeler: Skipping Swap Space (virtualMemory) as it matches zFileSystemMapIgnoreTypes.

2012-02-21 00:07:28,635 INFO zen.ZenModeler: Skipping Real Memory (ram) as it matches zFileSystemMapIgnoreTypes.

2012-02-21 00:07:28,636 INFO zen.ZenModeler: Skipping /proc/bus/usb. 0 total blocks.

2012-02-21 00:07:28,640 INFO zen.ZenModeler: Skipping Memory Buffers (other) as it matches zFileSystemMapIgnoreTypes.

2012-02-21 00:07:28,640 INFO zen.ZenModeler: Skipping /sys. 0 total blocks.

2012-02-21 00:07:28,643 INFO zen.ZenModeler: processing zenoss.snmp.UCDHardDiskMap for device 192.168.100.192

2012-02-21 00:07:30,295 INFO zen.ZenModeler: Changes in configuration applied

2012-02-21 00:07:30,377 INFO zen.ZenModeler: Scan time: 3.78 seconds

2012-02-21 00:07:30,389 INFO zen.ZenModeler: Daemon ZenModeler shutting down

RE: Distributed Collectors

community@zenoss.org — Tue, 21 Feb 2012 11:20:23 GMT

Hi Duc Le

The problem is that your RemoteCollector is trying to find the localhub using its domain name.

Try entering in the host fil on the RemoteCollector (normally in "/etc/hosts") and specify your localhubs ip to localhost.localdomain.

This could become a slight problem if bot the remotecollector and the localhub has the same domain name. ( if this is present, my advice would be to change one domain name)

Hope this helps

Cheers

RE: Distributed Collectors

community@zenoss.org — Wed, 22 Feb 2012 03:36:05 GMT

Hi Lumskekonig,

I have tried your advice and it works like a charm. Thanks much for your help.

Have a great day,

Duc

RE: Distributed Collectors

community@zenoss.org — Fri, 02 Mar 2012 15:19:10 GMT

Hi Guys,

Its time for extra security for me to put on collector. I want to enable firewall in collector. Have anyone already made the list of the ports to be opened for collector feature by looking at wireshark or any other means? If so, can you please share with us here.

Thanks & Regards.

RE: Distributed Collectors

community@zenoss.org — Fri, 02 Mar 2012 15:21:31 GMT

http://community.zenoss.org/docs/DOC-10246

RE: Distributed Collectors

community@zenoss.org — Thu, 22 Mar 2012 18:35:07 GMT

Hi Felipe,

I am going to upgrade to same set up as yours, Centos 6.2 and zenoss 3.2.0. I want to check does distributed collector works for you with this set up?

Thanks & Regards,

RE: Distributed Collectors

community@zenoss.org — Mon, 26 Mar 2012 10:09:49 GMT

Hi Egor & All,

Has anyone tested this ZenPack with Zenoss 3.2.1 (Current release)? I've tried with CentOS 5 (Zenoss as RPM package) and Ubuntu 10.04 (Zenoss as native DEB stack), with the same negative result, perhaps, either I missed importants steps or done not necessary extra ones...

My most important question -- on the 'slave', should I install Zenoss package and leave as it is, not touching it at all, or should I configure it first as 'normal' standalone Zenoss and then DistributedCollectors ZenPack will 'downsize' the slave to the necessary level, taking out non-needed daemons, etc... ?

SSH keys exchange is not an issue, Zenoss Central Hub is able to login to the Remote Collector machine via ssh using user 'zenoss' without password, but that's all. If I try to add 'Remote Monitor' on the Hub it accepts it, but nothing else happens, if I try to add device to remote collector, It won't be modelled...

Thanx in advance,

Kirill

RE: Distributed Collectors

community@zenoss.org — Mon, 26 Mar 2012 14:33:26 GMT

It works with 3.2.1. Yes, do an install on the collector and leave it as is, the ZenPack will take care of the configuration. You're missing setup steps for SSH.

According to what you've said, you've setup ssh keys such as is described here: http://community.spiceworks.com/education/projects/Passwordless_SSH_Using_Shared_Keys

The steps that you are missing are as follows (do these on the collector)...

As the zenoss user:

env > /home/zenoss/.ssh/environment

As root:

Edit /etc/ssh/sshd_config and add the following line to the end of it:

PermitUserEnvironment yes

Then restart sshd.

RE: Distributed Collectors

community@zenoss.org — Mon, 26 Mar 2012 15:14:38 GMT

Thanx a lot, Ryan, I'll try this right now... What is the reason behind this trick with 'env'? To keep strictly 'own local' environment of zenoss user on the slave and not infiltrate some stuff from central hub? OK, I need to finish install, store the environment and make 'diff' to see what this is all about...

RE: Distributed Collectors

community@zenoss.org — Mon, 26 Mar 2012 15:20:05 GMT

The reason for it is because the ZenPack is not creating a full ssh session then issuing commands. It is remotely issuing them one at a time such as...

ssh -l zenoss 'zenoss status'

SSH does not automatically inherit the environment for the zenoss user when commands are issued this way (as it does not run any of the user's login scripts which would result in the environment being set). The zenoss user environment is required to be able to issue zenoss commands. You have to therefore copy the environment in to /home/zenoss/.ssh/environment and configure sshd to allow the use of it.

RE: Distributed Collectors

community@zenoss.org — Mon, 26 Mar 2012 15:47:44 GMT

Aha, same old story, so-called No-TTY mode , thanx for tip...

RE: Distributed Collectors

community@zenoss.org — Thu, 26 Jul 2012 08:18:16 GMT

Hi Ryan,

I've follown the istruction but I'm stuck with the last step:

As the zenoss user:

env > /home/zenoss/.ssh/environment

I've got the Zenoss Virtual Machine, so I don't know the password of the zenoss user.... Maybe it's a newbie question, but how can I do?

Thanx

RE: Distributed Collectors

community@zenoss.org — Thu, 26 Jul 2012 14:30:14 GMT

The default root password is zenoss on the VM according to the documentation.

I'll also point out that it took me 1 minute of googling to find this.

RE: Distributed Collectors

community@zenoss.org — Thu, 26 Jul 2012 14:40:54 GMT

I think the question is what is the zenoss user password, not root, If that is actually the case, log in as root using the password Ryan has provided and then type `sudo su - zenoss` to become the zenoss user, without needing to know the zenoss user password.

RE: Distributed Collectors

community@zenoss.org — Thu, 26 Jul 2012 14:42:45 GMT

Fair point, he could have used passwd to change the zenoss password as root as well.

RE: Distributed Collectors

community@zenoss.org — Thu, 26 Jul 2012 15:17:22 GMT

dpetzel hit the point: I know the root password but I don't know the "zenoss" password. I didn't want to change it basically because I do not know how many config files (if any) I need to change after the password reset. I'm newbie about linux and I didn't know about the 'sudo su -zenoss'.

Now ssh seems to work correctly but when I create a device on the primary server given the secondary as collector, the job keeps running without any effect.

Any idea?

RE: Distributed Collectors

community@zenoss.org — Fri, 24 Aug 2012 17:37:36 GMT

Yes it does! I have Zenoss 3.2.0 running on CentOS 5.8 (stack-installer), with four collectors and a hub. Actually, I'm interested in upgrading my system to newer Zenoss 4.x (CentOS 6.3 rpms). Has anyone tried distributed collectors on the new Zenoss 4.x? There is a lot of new things on this new version... it seems great! Nevertheless it's really necessary to distribute my collectors... any clues?

RE: Distributed Collectors

community@zenoss.org — Sat, 25 Aug 2012 03:39:40 GMT

I'm pretty sure this isnt ready for Core 4 yet. I know I had taken a look at this during the beta, and there were some daemon names that are new in Core 4, that the distributed collector pack wouldnt support.

I've not heard one way or the other with certainty that it works or doesn't, but I'm pretty sure it doesn't (yet)

RE: Distributed Collectors

community@zenoss.org — Thu, 20 Sep 2012 20:18:26 GMT

Trying to set up two new 4.2 systems. When we add the remote we get this:

2012-09-20 12:19:51,000 INFO zen.updateCollector: Exception while performing command:

2012-09-20 12:19:51,000 INFO zen.updateCollector:

Traceback (most recent call last):

File "/opt/zenoss/Products/ZenUI3/browser/streaming.py", line 46, in __call__

self.stream()

File "/opt/zenoss/ZenPacks/ZenPacks.community.DistributedCollectors-2.0.egg/ZenPacks/community/DistributedCollectors/__init__.py", line 109, in stream

transaction.commit()

File "/opt/zenoss/lib/python/transaction/_manager.py", line 89, in commit

return self.get().commit()

File "/opt/zenoss/lib/python/transaction/_transaction.py", line 342, in commit

reraise(t, v, tb)

File "/opt/zenoss/lib/python/transaction/_transaction.py", line 333, in commit

self._commitResources()

File "/opt/zenoss/lib/python/transaction/_transaction.py", line 473, in _commitResources

reraise(t, v, tb)

File "/opt/zenoss/lib/python/transaction/_transaction.py", line 450, in _commitResources

rm.tpc_vote(self)

File "/opt/zenoss/lib/python/ZODB/Connection.py", line 781, in tpc_vote

s = vote(transaction)

File "/opt/zenoss/lib/python/relstorage/storage.py", line 794, in tpc_vote

return self._vote()

File "/opt/zenoss/lib/python/relstorage/storage.py", line 841, in _vote

serials = self._finish_store()

File "/opt/zenoss/lib/python/relstorage/storage.py", line 754, in _finish_store

rdata = self.tryToResolveConflict(oid, prev_tid, serial, data)

File "/opt/zenoss/lib/python/ZODB/ConflictResolution.py", line 277, in tryToResolveConflict

data=newpickle)

ConflictError: database conflict error (oid 0x13f0, class BTrees.OOBTree.OOBTree, serial this txn started with 0x039959b367c96844 2012-09-20 16:19:24.325019, serial currently committed 0x039959b3cde33c77 2012-09-20 16:19:48.254916)

Any pointers are greatly appreciated.

RE: Distributed Collectors

community@zenoss.org — Thu, 20 Sep 2012 20:22:57 GMT

Go in to zendmd and do...

reindex()

commit()

After that restart Zenoss and try again. Database conflicts like that can occur when the indexes are out of whack or the database is very busy.

RE: Distributed Collectors

community@zenoss.org — Thu, 20 Sep 2012 20:46:27 GMT

Thanks for the quick response.

Ran it and rebooted but get the same error.

Stupid question both servers show localhost is that the duplication?

Version is 4.2 on centos 6.3.

THX

RE: Distributed Collectors

community@zenoss.org — Thu, 20 Sep 2012 20:53:12 GMT

The other collector should not be named localhost, it should be named something else. The only collector named localhost should be the one on the master.

RE: Distributed Collectors

community@zenoss.org — Thu, 20 Sep 2012 22:21:38 GMT

Using the core install script rabbitmq fails to start if the name is not localhost.

RE: Distributed Collectors

community@zenoss.org — Thu, 20 Sep 2012 22:28:36 GMT

The way we handle a name other than localhost for rabbitmq is by doing...

echo 'NODENAME=rabbit@yourhostname' > /etc/rabbitmq/rabbitmq-env.conf

reboot

rabbitmqctl add_user zenoss zenoss

rabbitmqctl add_vhost /zenoss

rabbitmqctl set_permissions -p /zenoss zenoss '.*' '.*' '.*'

reboot

rm -f /var/log/rabbitmq/*zenoss-base*

rm -rf /var/lib/rabbitmq/mnesia/*zenoss-base*

The reboots probably aren't 100% necessary but we do it for good measure.

RE: Distributed Collectors

community@zenoss.org — Mon, 24 Sep 2012 20:37:20 GMT

I'm running 4.2 on main Zenoss and collector. I configured ssh public key authentication on the remote collector for user 'zenoss'. I can also ssh to the collector by command line successfully without a password. But I get the following error error when running the Add/Update Remote Collector portion of this Zenpack:

2012-09-24 16:30:37,000 INFO zen.updateCollector: Update Remote Collector remote.collector.com

2012-09-24 16:30:37,000 INFO zen.updateCollector: Stopping zenoss daemons

2012-09-24 16:30:37,000 INFO zen.updateCollector: Host key verification failed.

2012-09-24 16:30:37,000 INFO zen.updateCollector: Revert Remote Collector configuration

2012-09-24 16:30:37,000 INFO zen.updateCollector: Host key verification failed.

2012-09-24 16:30:37,000 INFO zen.updateCollector: Remove ZenPacks files from Remote Collector

2012-09-24 16:30:37,000 INFO zen.updateCollector: Host key verification failed.

2012-09-24 16:30:37,000 INFO zen.updateCollector: Copy ZenPacks files to Remote Collector

2012-09-24 16:30:37,000 INFO zen.updateCollector: Host key verification failed.

2012-09-24 16:30:37,000 INFO zen.updateCollector: cpio:

2012-09-24 16:30:37,000 INFO zen.updateCollector: write error

2012-09-24 16:30:37,000 INFO zen.updateCollector: : Broken pipe

2012-09-24 16:30:37,000 INFO zen.updateCollector: find:

2012-09-24 16:30:37,000 INFO zen.updateCollector: write error

2012-09-24 16:30:37,000 INFO zen.updateCollector: : Broken pipe

2012-09-24 16:30:37,000 INFO zen.updateCollector:

2012-09-24 16:30:37,000 INFO zen.updateCollector: Update Remote Collector configuration

2012-09-24 16:30:37,000 INFO zen.updateCollector: Host key verification failed.

2012-09-24 16:30:37,000 INFO zen.updateCollector: Starting zenoss daemons

2012-09-24 16:30:38,000 INFO zen.updateCollector: Host key verification failed.

2012-09-24 16:30:38,000 INFO zen.updateCollector: All Tasks Finished

I also ran "env > /home/zenoss/.ssh/environment" as zenoss user and restarted sshd with the option 'PermitUserEnvironment yes'. Is there anything else I'm missing? Is there any logging for the collectors to help troubleshoot?

RE: Distributed Collectors

community@zenoss.org — Mon, 15 Oct 2012 22:03:10 GMT

egor, I installed the github version of the pack and although everything works, my setup appears to confuse Zenoss. I use the remote collector to monitor vpn's using ping. I monitor the same vpn's from the main collector so to add them to the remote I fake the IP's to add them and then reset the IP to the real IP. This worked in 3.2.1. With 4.2.0 Zenoss always thinks that the event comes from the main collector so if I take down both vpn's I get 2 events generated for the main collector and none from the remote collector. Do you think this might be a problem with Zenoss or the Zenpack?

Thanks, Jim

RE: Distributed Collectors

community@zenoss.org — Wed, 07 Nov 2012 12:57:22 GMT

Hi all,

I have an issue where any devices moved to my remote collectors do not display graphs. Any ideas.

CentOS6.3 ZenOSS Core 4.2

RE: Distributed Collectors

community@zenoss.org — Wed, 07 Nov 2012 13:20:00 GMT

Check if zenrender daemon is running.

RE: Distributed Collectors

community@zenoss.org — Wed, 07 Nov 2012 17:35:24 GMT

Ok, zenrender was not running, I started it. My graphs are now showing but are empty.

I notice that the values for cur: avg: max: all show "nan"

I also noticed the following when I run a zenoss restart on the collectors:

zenoss restart
Daemon: zenrrdcached stopping...
Daemon: zenperfsql stopping...
already stopped
Daemon: zeneventlog stopping...
already stopped
Daemon: zenwin stopping...
already stopped
Daemon: zenwinperf stopping...
already stopped
Daemon: zenjmx stopping...
already stopped
Daemon: zenprocess stopping...
already stopped
Daemon: zencommand stopping...
already stopped
Daemon: zenperfsnmp stopping...
already stopped
Daemon: zenmodeler stopping...
already stopped
Daemon: zentrap stopping...
already stopped
Daemon: zenstatus stopping...
already stopped
Daemon: zensyslog stopping...
already stopped
Daemon: zenping stopping...
already stopped

RE: Distributed Collectors

community@zenoss.org — Wed, 07 Nov 2012 17:58:54 GMT

First off, don't use zenoss restart as it tends to cause problems. Do zenoss stop, then zenoss status to make sure everything is stopped, then zenoss start.

RE: Distributed Collectors

community@zenoss.org — Wed, 07 Nov 2012 18:14:57 GMT

OK so we learn. So after doing that and running the zenoss status again I get:

$ zenoss status
Daemon: zenping not running
Daemon: zensyslog not running
Daemon: zenstatus not running
Daemon: zentrap not running
Daemon: zenmodeler not running
Daemon: zenperfsnmp not running
Daemon: zencommand not running
Daemon: zenprocess not running
Daemon: zenjmx not running
Daemon: zenwinperf not running
Daemon: zenwin not running
Daemon: zeneventlog not running
Daemon: zenperfsql not running
Daemon: zenrrdcached program running; pid=4544

Seems that for some reason the Daemon's are not starting up.....Is there a logfile somewhere I can go look at.

RE: Distributed Collectors

community@zenoss.org — Wed, 07 Nov 2012 18:22:31 GMT

Yes, $ZENHOME/log has the logfiles for all the daemons.

RE: Distributed Collectors

community@zenoss.org — Wed, 07 Nov 2012 18:33:22 GMT

Don't I feel like the half idiot. I got the Daemons started. The collectors were trying to communicate with the Master by using the DNS name and there is no DNS record for the master. I added a record in the /etc/hosts file of each of the collectors and the daemons have started, however I still have no values except "nan" for all the counters this goes for the collector performance stats as well as for any devices I add/move to a collector.

RE: Distributed Collectors

community@zenoss.org — Wed, 07 Nov 2012 20:30:10 GMT

Since your collector wasn't able to connect to the master previously it never pulled down the configs for the devices and wasn't able to collect anything. You need to give it enough time to create the RRD files and capture data in them. If the devices already had RRD files on the master you can optionally copy those over to the same directory location on the collector.

RE: Distributed Collectors

community@zenoss.org — Thu, 08 Nov 2012 09:53:31 GMT

Thanks for that. I have received data now.

One other question. I have setup a "Template" collector and tried to use that as a deployment to the remote sites however when trying to add them as remote monitors I get an error about "serial currently committed", I can only assume that this is due to some serial number on the ZenOSS installation that is obviously now the same. Is it possible to change this serial number or do I have to load each remote collector from scratch.

RE: Distributed Collectors

community@zenoss.org — Thu, 08 Nov 2012 16:33:50 GMT

It's really recommended to just use a fresh install of Zenoss for collectors. This ZenPack takes care of configuring them provided you've setup the ssh keys and stuff as described on this page.

RE: Distributed Collectors

community@zenoss.org — Mon, 03 Dec 2012 10:05:25 GMT

Hi everyone.

I've been looking at this for a few hours and can't really get it working. I've set up the ssh key login bit, when I'm logged in on the master server as user zenoss I can ssh to the slave without password. So far good. I've also set the zeo.conf to "address 8100".

I was sort of expecting to be able to add the slave server on the master, get a few errors and then take it from there but when I go "add remote collector", type slave IP as ID and click OK the dialog closes and nothing happens.

Versions are: master, 3.1.0 and slave 3.2.1. Both servers are CentOS

Where should I start looking?

/Fredrik

RE: Distributed Collectors

community@zenoss.org — Mon, 03 Dec 2012 14:59:25 GMT

Did you complete the rest of the SSH setup steps?

Setup SSH keys between the master and the collector for the zenoss user like described here:http://community.spiceworks.com/education/projects/Passwordless_SSH_Using_Shared_Keys
As the zenoss user, issue the following command: env > /home/zenoss/.ssh/environment
As root, edit /etc/ssh/sshd_config and add the following line to the end of it: PermitUserEnvironment yes
Restart sshd.

RE: Distributed Collectors

community@zenoss.org — Tue, 04 Dec 2012 08:50:01 GMT

I've done those steps. The master server, as user zenoss, can log in to the slave server without password prompt.

RE: Distributed Collectors

community@zenoss.org — Tue, 04 Dec 2012 17:03:12 GMT

Can you do something like...

ssh -l zenoss yourcollector 'zenoss status'

You need to make sure that you're actually able to issue zenoss specific commands directly via ssh.

RE: Distributed Collectors

community@zenoss.org — Tue, 04 Dec 2012 17:15:32 GMT

Seems ok I get this when logged in as user zenoss on the master server

Daemon: zeoctl program running; pid=1393

Daemon: zopectl program running; pid=1398

Daemon: zenhub program running; pid=1437

Daemon: zenjobs program running; pid=1470

Daemon: zenping program running; pid=1533

Daemon: zensyslog program running; pid=1579

Daemon: zenstatus program running; pid=1574

Daemon: zenactions program running; pid=1607

Daemon: zentrap program running; pid=1722

Daemon: zenmodeler program running; pid=1704

Daemon: zenrender program running; pid=1735

Daemon: zenperfsnmp program running; pid=1768

Daemon: zencommand program running; pid=1802

Daemon: zenprocess program running; pid=1828

Daemon: zenwin program running; pid=1864

Daemon: zeneventlog program running; pid=1901

RE: Distributed Collectors

community@zenoss.org — Mon, 17 Dec 2012 21:07:42 GMT

Hi Ryan,

it doesn't look like my remote collector is getting the correct configuration for ie zenping.cnf is showing

hubhost localhost.localdomain instead of the correct IP or the name.

do i need to change all the cnf files on the remote collector to use the correct IP or name? Also where does that entry come from since if i update the remote collector all the settings gets changed again.

RE: Distributed Collectors

community@zenoss.org — Mon, 17 Dec 2012 21:13:41 GMT

The ZenPack is supposed to modify those configs for you provided that you've configured SSH on the collector correctly as described in the documentation on this page.

RE: Distributed Collectors

community@zenoss.org — Mon, 17 Dec 2012 21:21:43 GMT

the ssh works correctly. as zenoss when i SSH to the remote collector i dont get any password prompts. When i change all the .cnf files in /opt/zenoss/etc the modeling works. If i update the remote collector from the main it changes all that back to localhost.localdomain which is why i asked where is it getting this information from?

RE: Distributed Collectors

community@zenoss.org — Mon, 17 Dec 2012 21:29:27 GMT

I didn't write the pack but if I had to guess I'd say it's getting it from the hosts file on the collector.

RE: Distributed Collectors

community@zenoss.org — Mon, 17 Dec 2012 21:39:25 GMT

so with that said.. i ended up adding an alias to my /etc/hosts file on my main server:

main_server_name localhost.localdomain

so now when i update the collectors it still knows where to go to on the remote host. hope this helps others that have issues.

RE: Distributed Collectors

community@zenoss.org — Thu, 20 Dec 2012 18:51:51 GMT

So my remote collector is working but I dont have any graphs because the render URL is pointing to the remote collector. The problem i have is my remote collector is behind a firewall. We opened specific ports for communication to go back and forth Only on the master and the remote collector. Reading some information on port requirements there is a line that says:

"I found that using the distrubuted collectors ZenPack, the render URL is created to the remote collector on port 8091. What's more, clients actually make calls (when clicking on device graphs) to the render URL. IMHO this is a bug, but none the less, the client makes the calls. That requirement can be avoided by using HTTP-8090 to the ZenHub and let it proxy those requests over the persistence ZenHub connection that the collector will open to the ZenHub at startup"

Can someone point me in the right direction on how to set this up? I checked the Zenhub deamons and looked at the config file but am not really getting anywhere. Thanks.

RE: Distributed Collectors

community@zenoss.org — Thu, 10 Jan 2013 17:19:14 GMT

Hello all,

Has anyone tried the upgrade from 4.2.1 -> 4.2.3 while using this ZenPack? I'm wondering whether or not the fact that ZenOSS Core and ZenPacks are all in the same rpm will affect this. It seems as though the distributed collectors zenpack deletes all packs on the remote collecters and syncs again so that part should be ok, but after running the upgrade on dev, I see it re-installs all the zenpacks on a restart. I can forsee this being a possible issue and unfortunately my dev env is not distributed.

RE: Distributed Collectors

community@zenoss.org — Thu, 10 Jan 2013 17:22:56 GMT

It only reinstalls core packs, not community ones. It shouldn't hurt anything.

RE: Distributed Collectors

community@zenoss.org — Wed, 06 Feb 2013 22:17:13 GMT

Using 4.2.3 fresh install

I walked thru above guide to set up my first collector, kept me busy for the last week

Master and slave both identical installations Centos6.3 minimal
Installed zenpack distributed collector on master.
SSH public key authentication is set up SSH master to slave is no problem.
Added a collector, updated the collector (no errors) so sshing between master and slave should be fine.

Configured one node to be only accessible by the slave (remote collector) master tells me it is up but no ping and snmpwalk is ever done on that node. If i allow access on that node from master ip it runs everything.

Am i doing something wrong or is it no longer working for 4.2.3

Edit:

Googled a bit deeper and looked better in my logs.

Not only port 22 is used and a remote collector uses push events so i also need ports open at Master location. So a collector will push all his events to the master.

Correct me if im wrong (please don't shoot)

RE: Distributed Collectors

community@zenoss.org — Mon, 25 Mar 2013 13:55:48 GMT

I met the save problem with you : http://community.zenoss.org/message/72582

RE: Distributed Collectors

community@zenoss.org — Thu, 09 May 2013 20:27:00 GMT

Hey guys, I had the same issue with zooming from Chrome on ZenOSS 4.x (at least with our setup). I found that Chrome did not like the fact that the graphs were coming in insecurely when the rest of the page was coming in on 443. I was able to verify this by going to a device's graphs, clicking on the little shield in the URL field in Chrome and allowing all content. This allowed everything to zoom properly.

To resolve it, I put stunnel in front of zenrender. Here is my example config for stunnel:

# Managed by Puppet
pid = /var/run/stunnel.pid
cert = /etc/stunnel/stunnel.pem
foreground = no
syslog = no

[https]
accept = 8095
connect = 8091

I then setup a CNAME so we could use the same wildcard certificate for all the collectors and changed the zenrender urls under the collector settings:

https://zcol-co1.domain.net:8095

With this config in place, we are able to zoom in on graphs that reside on the remote collectors without issue.

Hope it helps

Matt

RE: Distributed Collectors

community@zenoss.org — Thu, 18 Jul 2013 11:20:04 GMT

I noticed a small bug when trying to upgrade distributed collectors, namely that I could not get zendmd to connect from collector to master. This was caused by an incorrent hostname in zodb_db_*.conf i /opt/zenoss/etc.

I looked in the zenpack and noticed that $ZENHOME/ZenPacks/ZenPacks.community.DistributedCollectors-2.0.egg/ZenPacks/community/DistributedCollectors/bin/updateConfigs had a typo in sed command:

sed -i 's/host .*$/host '$master'$/g' $ZENHOME/etc/zodb_db_main.conf

sed -i 's/host .*$/host '$master'$/g' $ZENHOME/etc/zodb_db_session.conf

Should be

sed -i 's/host .*$/host '$master'/g' $ZENHOME/etc/zodb_db_main.conf

sed -i 's/host .*$/host '$master'/g' $ZENHOME/etc/zodb_db_session.conf

This is a problem in 2.0 for the ZenPack.

RE: Distributed Collectors

community@zenoss.org — Wed, 24 Jul 2013 13:59:14 GMT

Getting this output when trying to install the remote collector:

2013-07-24 08:56:34,000 INFO zen.updateCollector: Add Remote Collector x1app014

2013-07-24 08:56:34,000 INFO zen.updateCollector: Stopping zenoss daemons

2013-07-24 08:56:35,000 INFO zen.updateCollector: bash: zenoss: command not found

2013-07-24 08:56:35,000 INFO zen.updateCollector: Remove ZenPacks files from Remote Collector

2013-07-24 08:56:35,000 INFO zen.updateCollector: Copy ZenPacks files to Remote Collector

2013-07-24 08:56:44,000 INFO zen.updateCollector: 115184 blocks

2013-07-24 08:56:44,000 INFO zen.updateCollector: Update Remote Collector configuration

2013-07-24 08:56:44,000 INFO zen.updateCollector: Starting zenoss daemons

2013-07-24 08:56:44,000 INFO zen.updateCollector: bash: zenoss: command not found

2013-07-24 08:56:44,000 INFO zen.updateCollector: All Tasks Finished

Almost like it is not using the zenoss user?

Not all the zenpacks on the 'master' are getting pushed either and only zencommand gets started on the remote collector after the initial add is complete?

Any ideas?
--Rob

RE: Distributed Collectors

community@zenoss.org — Thu, 01 Aug 2013 03:44:04 GMT

We're spec'ing out a datacenter which will have limited access. We're trying to discern the system requirements for the collector and have run into a few questions.

1) Does the collector forward the events to the main hub or does it store a local copy of the ZenEvents database?

2) Is the collector system Disk Space bound or CPU limited?

3) Generally speaking, if we want to monitor about 30 devices, what sort of system requirements should we be thinking of?

RE: Distributed Collectors

community@zenoss.org — Thu, 01 Aug 2013 13:04:32 GMT

1) The collector will forward the events to the main hub

2) In our collector setup, I would say its both. When you are using remote collectors all the RRD files for a device are stored on that collector (not on the main hub). The collector will run an instance of each of the daemons (zenperfsnmp, etc) so they will also be using a certain amount of CPU, but in my case I'd say disk space is the bigger limitation (disclaimer: we tweaked the default RRD settings to store finer grained data for longer, so our RRDs are bigger than the defaults).

3) Assuming you don't have an unusually high amount of data points on those 30 devices, I'd think you could get by with whatever the minimum requirements are for the version of Zenoss you intend to run

RE: Distributed Collectors

community@zenoss.org — Wed, 21 Aug 2013 20:01:57 GMT

I need some advice.

I had a working ZenOSS 3.2 deployment with two hubs used to monitor an internal service network plus switches, UPSs, etc and a remote hub monitoring more devices in what I can describe as a DMZ.

I set this up using the old manual instructions but there have been so many changes now this doesn't work for 4.2.4.

It used to work to have the zen daemons running out on the remote hub but configure zope to use mysql on the "main" collector.

This was nice because I could add devices out there and model, etc. but all my events and graphs, etc. would show up on the "main" hub.

I don't control the firewall and ports will not be opened for me; my old model worked because the connection came "from" the remote in the DMZ which was OK.

I just went through setting up the 4.2.4 remote using DistributedCollectors by creating the remote on my internal network and then placing it in the DMZ; I do seem to be getting events but the web front-end is gone. So now if I want to model or add new devices I guess I have to research the CLI zen commands for all of these which is a big step backwards; I would almost rather have two independant ZenOSSes.

Any ideas would be greatly appreciated.

RE: Distributed Collectors

community@zenoss.org — Thu, 22 Aug 2013 12:53:26 GMT

One practical note;

DistributedCollectors 2.0 added the monitor & hubhost values to my zredis.conf resulting in the error message:

[zenoss@myhost ~]$ /opt/zenoss/bin/zredis start starting... *** FATAL CONFIG FILE ERROR *** Reading the configuration file, at line 533 >>> 'monitor myhost.example.org' Bad directive or wrong number of arguments Usage: zredis[run|start|restart|stop|status|genconf] [options]

I commented these two lines out to resolve the issue.

RE: Distributed Collectors

community@zenoss.org — Thu, 22 Aug 2013 14:35:00 GMT

Yeah, zredis is new. Someone needs to update the pack to ignore it.

RE: Distributed Collectors

community@zenoss.org — Thu, 22 Aug 2013 17:38:50 GMT

My zen daemons are connecting to the zenhub service on the main hub fine and I do seem to be receiving events.

The zenpack updated the mysql DSN in the zodb_db_main.conf correctly and I verified the remote can make this connection if needed.

Does my remote hub need to run a rabbitmq-server or should I try to configure it to connect to the rabbitmq-server on the main hub?

What do I need to do to get the web interface running again?

I did have the rrd graph problem but this went away after configuring the renderurl; I thought the zenpack was supposed to do that for me but no matter.

RE: Distributed Collectors

community@zenoss.org — Thu, 12 Sep 2013 11:53:19 GMT

Have everything working. Thanks everyone.

One issue:

My RRD files are being created on the master. Shouldn't they be on the collector?

Jeff

RE: Distributed Collectors

community@zenoss.org — Fri, 13 Sep 2013 13:23:56 GMT

Hi,

I used this zenpack to install 3 new collectors in a new 4.2.4 setup. The collectors seem to be setup correctly but I cannot seem to get the performance graphs to work. They show up with NAN. These new collectores have been up for a little more than a week so the graphs should have created by now. My render URL is http://collector_hostname:8091. I have configured and started zenrender on the collector and master with the right conf files. I can telnet from the master to port 8091 on the collector. All of the zenoss logs and /var/log/messages (as it seems zenrrdcached logs there) show no errors.

I tried removing all the directories and files under /opt/zenoss/perf/Daemons on the collector and they re-appeared under localhost directory. On old version of zenoss where collector performance graphs are working the RRDs are located on the collector under /opt/zenoss/perf/Daemons/Collector_hostname.

Can someone explain more what actually creates the daemon graphs? Is it zenstatus?

What can I do next to troubleshoot?

RE: Distributed Collectors

community@zenoss.org — Fri, 13 Sep 2013 13:38:19 GMT

Jeff, did you create the collector as "add monitor" or "add remote monitor"? I noticed that the difference between these two seems to be the location of (amongst other things) the RRD performance files.

This difference is not very clear, and I'm not sure what the exact difference between "monitor" and "remote monitor" is... but I think that you'd like to use the remote monitor.

Thomas

RE: Distributed Collectors

community@zenoss.org — Fri, 13 Sep 2013 14:39:27 GMT

Thomas,

I think I added it before using "add monitor". I removed the collector then added it using "add remote monitor". The RRD files are being created on the collector now. Thanks!

Now my graphs aren't rendering..... yay.... I know I saw a post for that somewhere....

Thanks again!

Jeff Creek

Network Systems Administrator

2101 E. Coliseum Boulevard

Fort Wayne, Indiana 46805-1499

Phone: (260) 481-5757

E-mail: creej@ipfw.edu

>>> thomas <community@zenoss.org> 9/13/2013 9:38 AM >>>

Zenoss Community

Distributed Collectors

new comment by thomas View all comments on this document

This difference is not very clear, and I'm not sure what the exact difference between "monitor" and "remote monitor" is... but I think that you'd like to use the remote monitor.

Thomas

Reply to this email to respond to thomas's comment.

RE: Distributed Collectors

community@zenoss.org — Sat, 14 Sep 2013 19:42:10 GMT

kenhen93: try changing the render url of the collector from its hostname to the ip address.

advanced > collectors > collector_name > edit > render URL

also check zenoss status: zenrender should be running in remote collector.

RE: Distributed Collectors

community@zenoss.org — Mon, 16 Sep 2013 11:45:32 GMT

I set the render url to proxy through the master. http://zenossMaster:8090/zenossCollector

So, (I believe) the master is doing the rendering because zenrender is not running on the collector and the graphs display properly.

Jeff Creek

Network Systems Administrator

2101 E. Coliseum Boulevard

Fort Wayne, Indiana 46805-1499

Phone: (260) 481-5757

E-mail: creej@ipfw.edu

>>> ignacio freyre <community@zenoss.org> 9/14/2013 3:42 PM >>>

Zenoss Community

Distributed Collectors

new comment by ignacio freyre View all comments on this document

kenhen93: try changing the render url of the collector from its hostname to the ip address.

advanced > collectors > collector_name > edit > render URL

also check zenoss status: zenrender should be running in remote collector.

Reply to this email to respond to ignacio freyre's comment.

RE: Distributed Collectors

community@zenoss.org — Mon, 16 Sep 2013 13:36:58 GMT

thats good to know, nice alternative so you wont have to acces collectors directly but through the master instead.

RE: Distributed Collectors

community@zenoss.org — Mon, 23 Sep 2013 20:05:47 GMT

I used this zenpack to install my collectors but when i try to run zenperfsnmp run -d <device> -v10 on a device that is linked to the collector I get the following error

ERROR zen.collector.config: Configuration for <server> unavailable -- is that the correct name?

Any idea what is going on here? I tried deleting and re-adding the device, same problem.

>>> d=dmd.Devices.findDevice('collector')

>>> d.__class__

RE: Distributed Collectors

community@zenoss.org — Tue, 24 Sep 2013 06:53:04 GMT

When a device is monitored by a remote collector, you cannot run a command towards that device from you main zenoss server.

If you need to invoke ie. zenperfsnmp towards a device monitored by a remote collector, you need to run zenperfsnmp on that remote collector.

(this is what I have discovered... if someone knows how to trigger a run, I'd be happy to know)

RE: Distributed Collectors

community@zenoss.org — Tue, 24 Sep 2013 10:14:57 GMT

Just ssh + command!

Em terça-feira, 24 de setembro de 2013, thomas escreveu:

**
Zenoss Community <http://community.zenoss.org/index.jspa> Distributed Collectors new comment by
thomas <http://community.zenoss.org/people/thomas> View all comments on
this document <http://community.zenoss.org/docs/DOC-5861#comment-6864>

RE: Distributed Collectors

community@zenoss.org — Tue, 24 Sep 2013 10:17:21 GMT

I mean, create a command on advanced tab, for example "pingFromCollector",

copy the hub Ping command line, and add "ssh" on the beggining!

Em terça-feira, 24 de setembro de 2013, Felipe Castelar escreveu:

Just ssh + command!

Em terça-feira, 24 de setembro de 2013, thomas escreveu:

>> **

>> Zenoss Community <http://community.zenoss.org/index.jspa> Distributed Collectors new comment by

>> thomas <http://community.zenoss.org/people/thomas> View all comments on

>> this document <http://community.zenoss.org/docs/DOC-5861#comment-6864>

RE: Distributed Collectors

community@zenoss.org — Tue, 24 Sep 2013 12:43:30 GMT

Thomas, I am running the command from the collector. Where do the collectors get their configuaration information? The collector properly connects to zenhub on my master server.

RE: Distributed Collectors

community@zenoss.org — Tue, 24 Sep 2013 12:57:34 GMT

Uhmm, ok, I thought you ran the command on the zenoss-master and not the remote collector.

The remote collector fetches information by connecting to your zenoss-master's zope-db which holds configurations.

But I was wondering, can you connect to zodb from you collector? You should be able to login to your remote collector and start zendmd as the zenoss user. From here you should be able to do a:

d=find('device-name')

The d-object should then be your device.

RE: Distributed Collectors

community@zenoss.org — Tue, 24 Sep 2013 13:48:36 GMT

Thomas, I can't see to do that as it complains mysql is not running but I don't think that should work. Mysql should not be started on a collector

RE: Distributed Collectors

community@zenoss.org — Wed, 25 Sep 2013 05:39:33 GMT

Hmm, check your configuration in /opt/zenoss/etc/global.conf and others maybe, to see if it has not been provisioned correctly by the zenpack.

I have the following variables set in different files to point to my zenoss master:

global.conf	zodb-host zenoss-master
zencommand.conf	hubhost zenoss-master
zeneventlog.conf	hubhost zenoss-master
zenjmx.conf	hubhost zenoss-master
zenmodeler.conf	hubhost zenoss-master
zenperfsnmp.conf	hubhost zenoss-master
zenping.conf	hubhost zenoss-master
zenprocess.conf	hubhost zenoss-master
zenrrdcached.conf	hubhost zenoss-master
zenstatus.conf	hubhost zenoss-master
zensyslog.conf	hubhost zenoss-master
zentrap.conf	hubhost zenoss-master
zenwin.conf	hubhost zenoss-master
zenwinmodeler.conf	hubhost zenoss-master
zenwinperf.conf	hubhost zenoss-master
zodb_db_main.conf	host zenoss-master
zodb_db_session.conf	host zenoss-master

Maybe this is the problem? Or maybe a firewall issue between you master and collector.

RE: Distributed Collectors

community@zenoss.org — Mon, 11 Nov 2013 10:02:27 GMT

We have multiple sites with servers thats have identical IP schemes (due to a 3rd party system that we are unable to change)

From my understanding this Zen pack will allow the remote collectors to link up to the master, but wont allow you to use the same IP for a device twice, even if its on different collectors?

RE: Distributed Collectors

community@zenoss.org — Mon, 11 Nov 2013 14:15:24 GMT

That is correct. The ability to do what you describe is only available in the Enterprise version of Zenoss.

RE: Distributed Collectors

community@zenoss.org — Tue, 19 Nov 2013 02:55:52 GMT

how do you really fix this issue ?

*** FATAL CONFIG FILE ERROR ***

Reading the configuration file, at line 533

>>> 'monitor 10.231.12.164'

Bad directive or wrong number of arguments

Daemon: zeneventlog /opt/zenoss/bin/zenoss: line 115: /opt/zenoss/bin/zeneventlog: Permission denied

Daemon: zenwinperf /opt/zenoss/bin/zenoss: line 115: /opt/zenoss/bin/zenwinperf: Permission denied

Daemon: zenwin /opt/zenoss/bin/zenoss: line 115: /opt/zenoss/bin/zenwin: Permission denied

Thanks.

RE: Distributed Collectors

community@zenoss.org — Tue, 19 Nov 2013 07:26:56 GMT

Take a look here:

https://github.com/epuzanov/ZenPacks.community.DistributedCollectors/pulls

There is a pullrequest to solve this issue, If you checkout the source and ad these changes, then you should be good to go.

RE: Distributed Collectors

community@zenoss.org — Tue, 19 Nov 2013 19:36:43 GMT

Thanks Thomas,

How do I disable it, its breaking my start up.

Thank you

RE: Distributed Collectors

community@zenoss.org — Thu, 13 Mar 2014 14:32:52 GMT

Is there a way to copy over the SSH key in ubuntu without a zenoss password?

Have tried on the master server

ssh-keygen -t rsa

cat ~/.ssh/id_rsa.pub | ssh zenoss@remote_collector_name "cat - >> ~/.ssh/authorized_keys"

but keep getting asked for the zenoss password on the remote server

RE: Distributed Collectors

community@zenoss.org — Thu, 13 Mar 2014 14:39:47 GMT

Why wouldn't you just copy it over as root and adjust the permissions afterwards? You don't need to use the zenoss user, just get it copied over whatever way you can.