ZenPacks : Formula Data Source : Comments

RE: Formula Data Source

community@zenoss.org — Wed, 01 Dec 2010 19:50:18 GMT

Cool that you got a new version with support for more templates out there. Im not quite sure what this means? Did it only support Data Sources from other templates before?

RE: Formula Data Source

community@zenoss.org — Wed, 01 Dec 2010 19:54:54 GMT

It only supported Data Sources from Device templates before (not interface, process, or filesystem templates).

For example, if you go in to a Device template, you see a Target Class of Products.ZenModel.Device. This means that data from datapoints in that template can be gathered by doing something like:

dmd.Devices.findDevice(‘Devicename’).getRRDValue(‘datapointname’).

In the case of an interface template which has a Target Class of Products.ZenModel.IpInterface you have to do things a little differently to get the data. You would do something like:

d = dmd.Devices.findDevice('Devicename')
for i in d.os.interfaces():
if “interfacename” == i.id:
data = i.getRRDValue(‘datapointname’)

I adjusted the datapoint script so that it passes both the name of the component (whether it be the name of the interface, process or filesystem), as well as the Target Class for the template to the script that actually gathers the data and performs the calculation.

Here’s a section of code from the script as an example:

if options.target == "Products.ZenModel.IpInterface":
    d = dmd.Devices.findDevice(options.device)
    for q in d.os.interfaces():
        if options.object == q.id:
            i = q
    formula = re.sub(r"\%(\w*)", r"i.getRRDValue('\1')", options.formula)
    formula = re.sub(r"\here.(\w*)", r"i.\1", formula)
    eval(formula)

I’ve yet to come up with a way to do this dynamically. I need to figure out if there’s some function that I can use to determine the location that I need to query for RRD data at based on the template type. For example, a way to know that I need to query data at d.os.interfaces() for Products.ZenModel.IpInterface. So far I haven’t found a way, but I intend to keep working on it.

RE: Formula Data Source

community@zenoss.org — Tue, 26 Apr 2011 20:00:28 GMT

The ZenPack now supports all templates regardless of type.

RE: Formula Data Source

community@zenoss.org — Wed, 27 Apr 2011 07:49:25 GMT

First of all I want to say You a big THANKS for this ZenPack. I have several cases where I have to calculate a value to monitor.

But I have problem with performance when using it. It just kills CPU. I have Zenoss 3.1.0 Core on CentOS 5.5 64 bit.

Now I see that you've removed plans to rewrite it as a daemon. Does it mean that you've stopped the development of this ZenPack?

RE: Formula Data Source

community@zenoss.org — Wed, 27 Apr 2011 15:42:47 GMT

No, I just removed that to clean up the look of this page. Future performance enhancements are still planned. Also, version 1.6 should be lighter on CPU than the previous version was.

RE: Formula Data Source

community@zenoss.org — Thu, 19 May 2011 14:52:46 GMT

I just installed this yesterday, it's exactly what I was looking for (calculating some data points which were done internally in Cacti using other gathered SNMP data). However just as Tornado mentioned I see the CPU usage rose dramatically after installing:

It seems pretty constant that the machine is running the calc_formula.py script, I assume once per formula per machine which needs it (which in our case is twice for just about every machine). I'm guessing a daemonized version of this will make that much faster?

RE: Formula Data Source

community@zenoss.org — Thu, 19 May 2011 14:59:57 GMT

That's completely expected with this version. The ZenPack currently launches a script for each calculation. This means that there are a bunch of scripts (threads) running concurrently which causes heavy load on the CPU. As I've stated, I will be writing a daemon for the pack eventually which will allow all of the calculations to be done under a single thread, which will improve performance. I'm completely *swamped* at work right now, so I have no idea when I'll actually get around to coding the daemon. Coding a Zenoss daemon is quite time consuming. I personally only use the pack on a few very small device groups, so it works fine as is. I wouldn't recommend using it in templates such as interface or filesystem at the moment. I'll see if I can find some time to start on it in a couple of weeks when things at work will have hopefully died down.

RE: Formula Data Source

community@zenoss.org — Wed, 25 May 2011 11:42:00 GMT

Hi Ryan

i just started to use your Formula Datasource zenpack, however I don't quite understand how to reference existing datasources/datapoints (since you repeat similar names in your example and i can't say which is which).

To simplify, let's ay i already have some datasources (DSource) and datapoints (DPoint) created:

DSource1

DPoint1
DPoint2

DSource2

DPoint3
DPoint4

Let's say i want to calculate DPoint2 + DPoint4.

I create a new DATASOURCE for my formula, "FormulaDSource" and it automatically creates a corresponding GAUGE type Datapoint "FormulaDPoint" (names are identical). So now i have these:

DSource1

DPoint1
DPoint2

DSource2

DPoint3
DPoint4

FormulaDSource

FormulaDPoint

Upon editing the FormulaDSource, i have to enter my (DPoint2 + DPoint4) calculation in the Data Formula field. What would be the correct entry:

1. $DSource1_DPoint2 + $DSource2_DPoint4

2. $DPoint2 + $DPoint4 ????

Actually, I tried both options but my formula data source is still not producing a calculation (required datapoints for calculation are okay and are returning valid values)

PS. one of my datapoints is named like "Datapoint_abc" (has to be like that) so maybe referencing as $Datasource_Datapoint_abc (with two underscores) is causing the problem?

thanks

marko

RE: Formula Data Source

community@zenoss.org — Wed, 25 May 2011 13:41:37 GMT

You are correct in your assumption that it's Datasource_Datapoint. They have to be EXACTLY what you set them as. If your datapoint is "Datapoint_abc" and your datasource is "Datasource", then you would use $Datasource_Datapoint_abc. It should work. The pack uses Zenoss' standard getRRDValue routine. Have you tried running a test against the datasource?

RE: Formula Data Source

community@zenoss.org — Thu, 26 May 2011 08:30:49 GMT

Thanks Ryan for confirming this.

No, my only Fromula DS experiment was on a component template (and we all know component template datasource tests usually fail due to the dynamic parameters required there - you can only test against a device, not the device AND a component).

I will repeat my experiment on some device template and post my results here.

regards

marko

RE: Formula Data Source

community@zenoss.org — Thu, 26 May 2011 13:50:14 GMT

If you're testing against a component then from the commandline as the zenoss user you can do: zencommand run -v10 -d <devicename> to test. Eventually when I implement a daemon for the pack you'll do the same thing with that daemon.

RE: Formula Data Source

community@zenoss.org — Fri, 27 May 2011 07:47:34 GMT

Ryan,

i made a formula Datasource in a device template, added (+) two existing datasources for test and it works, i understand the process now.

It's kinda slow (the calculation) so i had to increase both zCommandCycle and zCommandtimeout to get it calculated in time and graphed properly (without any zCommand timeout events)

thanks for your help

marko

RE: Formula Data Source

community@zenoss.org — Fri, 27 May 2011 18:51:15 GMT

It'll be slow until I code the daemon for it.

RE: Formula Data Source

community@zenoss.org — Thu, 27 Oct 2011 16:37:03 GMT

I'm having issues with this on 3.2.1 in relation to the edit data source.

This is what I see instead of the one shown above.

RE: Formula Data Source

community@zenoss.org — Thu, 27 Oct 2011 16:53:32 GMT

They must have changed the way the UI components are coded yet again. I'll fix it when I have some spare time (not sure when that'll be).

RE: Formula Data Source

community@zenoss.org — Thu, 27 Oct 2011 20:14:05 GMT

I just experienced the same issue as zoomequipd.

Just curious if you have already fixed this.

Thanks.

RE: Formula Data Source

community@zenoss.org — Thu, 03 Nov 2011 21:19:38 GMT

Hey Ryan,

I had a couple questions about this ZenPack I was hoping you could help with.

1. I wasn't able to use a datasource/datapoint until I added it to a graph. If I understand correctly it's because that's where you get the value using getRRDValue (or something to that effect).

Would it be relatively easy to just take the output of the datasource instead of getting it from the RRD?

2. I'm using remote/multiple collectors and I've noticed the formula datasource doesn't work for my devices on the remote collectors because they look like they're trying to query localhost:8100 when zope is running on another server.

Same question as above. Would it be relatively easy to update the code so it checks $ZENHOME/etc/zope.conf for the zope db location?

Thanks!

RE: Formula Data Source

community@zenoss.org — Thu, 03 Nov 2011 21:25:54 GMT

1. No you don't need to add the datasource to a graph, it just needs to be present in the template. It needs to have actually been there for 3 collection cycles before getRRDValue will even work.

2. With it's current design the getRRDValue does get executed on the remote collector instead of the master server. It doesn't currently work with collectors.

To completely optomize this thing a daemon needs to be coded that only runs on the master server and which executes getRRDValue for each datapoints, executes the formula, and then creates/writes to the RRDs. This daemon is nothing like any of the daemons that have been coded so far so it's going to be a huge challenge. I've just been too busy at work and in general to put the time in to doing this.

RE: Formula Data Source

community@zenoss.org — Fri, 10 Feb 2012 19:56:16 GMT

Ryan,

I'm attempting to use this to calculate a ratio between to absolute datatypes. When I test it against a device, I get a TypeError

-----8<-----8<-----

Preparing Command...

Calculating formula '$lbvserver_lbvserver_responsesrate / $lbvserver_lbvserver_requestsrate' against 10.255.1.51

The formula generated an error: TypeError.

DONE in 3 seconds

----->8----->8-----

Any Ideas?

--Zenoss, version 3.2.1 - CentOS 5.6 - Dell PowerEdge 1950

RE: Formula Data Source

community@zenoss.org — Mon, 13 Feb 2012 19:27:20 GMT

The pack uses the getRRDValue function to gather the data. Try doing the following in zendmd...

d = dmd.Devices.findDevice('yourdevicename')

d.getRRDValue('lbvserver_lbvserver_responsesrate')

d.getRRDValue('lbvserver_lbvserver_requestsrate')

See if it returns values.

RE: Formula Data Source

community@zenoss.org — Mon, 13 Feb 2012 19:46:29 GMT

That's the problem. Since there is only one managment IP address for the Netscaler and all the virtual servers. the perf data is all associated with the First defined device, the Netscaler.

It looks like I need to change how I create virtual servers devices in zenoss

Thanks for the clue.

RE: Formula Data Source

community@zenoss.org — Thu, 08 Mar 2012 23:04:14 GMT

Hi Ryan,

Thanks for implementing this Zenpack, it really helps my work.

However a quick questio:

When I was calculating values, I found if I encounter 0 division problem (anynumber / 0), the script just throws an exception(I guess it was thrown by Python eval method), which results an warning in Zenoss: "No data returned".

Would it be better if the script can return 0 or NaN in this case?

Thank in advance.

RE: Formula Data Source

community@zenoss.org — Fri, 09 Mar 2012 17:05:56 GMT

I'll fix that in the next version (it could be a while before I have time to do an update).

RE: Formula Data Source

community@zenoss.org — Fri, 09 Mar 2012 19:25:10 GMT

Thanks Ryan, looking forward to the new version!

RE: Formula Data Source

community@zenoss.org — Fri, 20 Apr 2012 18:16:32 GMT

Ryan,

This zenpack looks like I can use different datasources & datapoints from the same device...correct?

Can I manipulate the same datasource/datapoint across different devices? If I understand this correctly the answer is no?

RE: Formula Data Source

community@zenoss.org — Fri, 20 Apr 2012 18:20:53 GMT

The pack currently doesn't support that. Only datasources on a single device.

RE: Formula Data Source

community@zenoss.org — Fri, 20 Apr 2012 18:27:57 GMT

k thank you!

RE: Formula Data Source

community@zenoss.org — Tue, 19 Jun 2012 15:13:22 GMT

Ryan, I had a strange issue with this zenpack and thought I would at least report it. I added a formula graph to a template, about 20 devices used this template. it caluclates perfectly. What was odd is 17 out of the 20 devices started graphing the output. The 3 that didn't I looked into the /Devices dir and no rrd file was created. I tried remodeling it etc. No matter what I did through the gui I couldnt get it created.

Cutting to the chase I ended up just touching/creating the file in the Device dir and these 3 immediately starting creating graph points. There is no pernissions issues with the zenoss id etc.

It is working fine now.

Initially I had a timeout issue which had to be increased for the 17/20 to work.

End of my 2 cents...

thanks again for the pack!

This was on v3.2.1 on cent os.

Jay

RE: Formula Data Source

community@zenoss.org — Tue, 19 Jun 2012 15:19:43 GMT

Hmmm, I've never heard of that issue before. I would have needed output from a "zencommand run -v10" to be able to comment on what the cause might have been for that.

RE: Formula Data Source

community@zenoss.org — Wed, 27 Jun 2012 22:27:12 GMT

Hi Ryan,

I'm a summer intern and fairly new to Zenoss. I've been trying to use your ZenPack to get the difference of two data points and it seems to work fine when I test it from the Edit Data Source window, but is the result value automatically kept as a new data point? I've added your formula data source to my template, named it, entered a formula, enabled it, and saved it. So now do I just create a data point for the data source and name it whatever I choose? Would you please clarify this for me? Thank you.

RE: Formula Data Source

community@zenoss.org — Thu, 28 Jun 2012 14:23:32 GMT

I coded the pack so that it creates both the datasource and the datapoint. All you need to do is create the datasource, add your formula, and then add it to a graph as a graphpoint. Everything else should be automatically taken care of.

RE: Formula Data Source

community@zenoss.org — Thu, 28 Jun 2012 17:45:34 GMT

Thanks. That is what I'd expected. However, I'm using an enterprise edition of zenoss for which the core code has been modified significantly, so I think it may be interfering with your zenpack. The data point doesn't seem to be being created, or, if it is, it isn't apparant. What is the standard way that the data point is named? Thank you.

RE: Formula Data Source

community@zenoss.org — Thu, 28 Jun 2012 18:00:34 GMT

The enteprise Zenoss is currently version 4. Since they released version 4 of enterprise before releasing version 4 of core, the community hasn't had a chance to update any of their packs to work with the new architechture. I'm waiting for Core 4 to be released to update any of my ZenPacks that need updating.

RE: Formula Data Source

community@zenoss.org — Tue, 14 Aug 2012 15:43:07 GMT

I tested this pack in Core 4.2 yesterday and it worked without any issues.

RE: Formula Data Source

community@zenoss.org — Tue, 14 Aug 2012 20:25:05 GMT

Hi Ryan,

Yes, I apologize, my problem was short lived - I must have made an error,

perhaps not pushing the changes.

Thank you.

RE: Formula Data Source

community@zenoss.org — Mon, 20 Aug 2012 19:18:50 GMT

This version 1.6 works in Zenoss 3.2.1?

In my tests, with data point generate by nagios plugin "check_snmp" didn't work.

Thanks

RE: Formula Data Source

community@zenoss.org — Mon, 20 Aug 2012 19:20:26 GMT

It works in Zenoss 2, 3, and 4. Don't put spaces in your datasource names, it won't work if there are spaces in them.

RE: Formula Data Source

community@zenoss.org — Mon, 20 Aug 2012 19:25:16 GMT

Example:

Data Source:

cg.NNC-0200

cg.NNC-0210

Data Point:

nnc.110.9.1

nnc.110.9.2

Data Formula:

$cg.NNC-0200_nnc.110.9.1 + $cg.NNC-0210_nnc.110.9.2

Is this correct?

Thanks

RE: Formula Data Source

community@zenoss.org — Mon, 20 Aug 2012 19:26:21 GMT

Correting:

$cg.NNC-0200+$nnc.110.9.1

RE: Formula Data Source

community@zenoss.org — Mon, 20 Aug 2012 19:29:49 GMT

It's possible that the periods or dashes are throwing it off. Also, put some spacing between the +. First off try...

$cg.NNC-0200 + $nnc.110.9.1

Notice the spacing. If that doesn't work then rename your datasources and try...

$cgNNC0200 + $nnc11091

It sometimes doesn't play well with special characters, something that I need to work on.

RE: Formula Data Source

community@zenoss.org — Mon, 20 Aug 2012 19:34:10 GMT

I'll do the tests.

thanks

RE: Formula Data Source

community@zenoss.org — Mon, 20 May 2013 12:03:55 GMT

Dear all,

After i installed this very handy zenpack, in my localy Defined HttpMonitor Template , i added a new FORMULA type DataSource and named it TS.

Normally it automatically creates a corresponding GAUGE type Datapoint "TS.TS"

Now my Template look likes below :

DSource1

DPoint1
DPoint2

DSource2

DPoint3
DPoint4

FormulaDSource //TS

FormulaDPoint // TS.TS

I double clicked on FormulaDPoint (here as TS.TS) and now i dont know where to write my Formula ?!

In CREATE COMMAND or ID/FORMULA or RRD Minimum Maximum ?

My formula is $DSource1_DPoint2 + $DSource2_DPoint4

RE: Formula Data Source

community@zenoss.org — Tue, 21 May 2013 18:45:26 GMT

Double click on the datasource, not the datapoint. The formula always needs to be set on the datasource. Your formula looks fine.

RE: Formula Data Source

community@zenoss.org — Wed, 04 Sep 2013 01:42:27 GMT

Hi Ryan,

Is it possible to make the addition of two different devices and show the result in a Graph, for example:

Device A interface eth0 + Device B interface eth0.

And see the total amount of bandwidth used in a Graph ??

Thanks in advance

RE: Formula Data Source

community@zenoss.org — Wed, 04 Sep 2013 13:43:02 GMT

That is not currently possible with this pack, you'd have to script something like that from scratch.

RE: Formula Data Source

community@zenoss.org — Thu, 12 Dec 2013 22:07:00 GMT

When attempting to edit the data source, I don't get the "Data Formula" option. I see the below. This is in Zenoss 4.2.4. Any ideas?

RE: Formula Data Source

community@zenoss.org — Wed, 18 Dec 2013 18:59:28 GMT

Try installing the ZenPacks.community.FormulaDataSource-1.6-py2.7.egg.zip and see if that makes a difference. I just quickly tested that file on my 4.x instance and I could see the field. Make sure you restart Zenoss after installing the pack.

RE: Formula Data Source

community@zenoss.org — Fri, 20 Dec 2013 18:05:08 GMT

thank you, that seemed to have helped.