Beta AIX zenpack for Beta Zenoss

Up to Discussions in zenoss-dev

1 2 3 4 Previous Next 112768 Views 56 Replies Latest reply: Jan 26, 2009 10:28 AM by Juerg Gerber

kpg123

157 posts since
Jun 29, 2007

Currently Being Moderated

May 9, 2008 5:19 PM

I've just added a beta version of an AIX Zenpack for the beta version of Zenoss. Perhaps "beta" is too strong of a recommendation for this ZenPack -- all I will guarantee is that it won't end all life on the planet (okay, maybe just Earth is safe... :)

This is an egg-style zenpack, so it most assuredly won't work with 2.1.3.

http://www.zenoss.com/community/wiki/user-contributed/

**Use this ZenPack only with a test Zenoss system to make sure that there's no possibility of messing up your other performance data.**

First, the bad news...

Known issues:
* requires SNMP version 3 with authentication off (ie SNMP v1 commands will work with this feature set). Description of modifications required in the description for the /devices/servers/AIX class.
* doesn't properly map to the manufacture links for any of the hardware found (eg disk, cpus)
* doesn't allow for re-exports of the ZenPack
* the performance collectors have no error checking and are written in bash and perl (Did I mention the word "beta" yet?)
* the performance collectors explicitly require that version 1 snmpwalk and snmpget work, and won't accept other credential info
* filesystem events don't insert the name of the filesystem
* sanity checks in the modelers seem to want too much sanity, as they *seem* to sometimes prevent modeling (aka discovery) from gathering all information. On the plus side, the sanity checking does prevent Zenoss from blowing away previous modelled device info when it can't talk to the SNMP agents or the machine is down. :)
* the AIXBase modeler still doesn't seem to properly identify the hardware model and update the correct spot for the hardware. The serial number does appear to be updated correctly for the hardware, however.
* doesn't show the per-processor statistics, only a total averaged number. In addition, the MIB doesn't update the statistics all that frequently (I might have missed a setting that can up the polling on CPU usage), so it may even be fantastically wrong.

.. and now the good news:

Features:
* uses standard IBM AIX snmpd services. No net-snmp daemons required, no other packages to install.
* adds subclasses for /devices/servers/AIX, with further subclasses for LPARs and VIO servers. Adds a token link into /devices/server/Linux/HMC with a customized zLink so that it's (perhaps?) easier to get to the HMC console.
* (YMMV) discovers memory, paging space, processors, disks, adapters and filesystems. NB: FRU information and model numbers would (I *think*) require updates to other ZenModel relationships, which would need to be in the base Zenoss.
* has a copy of the AIX MIB (from the samples directory) and cleaned up version of the xmtopas-generated MIB (the MIB it generates is, to be polite, syntactically challenged. It may also be plain wrong in places. :)
* allows for a lot more performance statistics to be gathered

While fixes are especially welcome, other feedback is also appreciated. :)

Thank you,

kells

Future possibilites:
Note that storage-related info such as VG + LV + filesystem + MPIO path + LUN ID mapping will require ZenModel relationships to be created, which will require time. This would be useful to be able to create a report to determine (for example) which filesystems would be affected by an outage of which SAN storage system. Some info may also not be available through SNMP.

The VIO mapping of virtual adapters to physical adapters and LUN assignments might be impossible through SNMP. I have written Perl scripts to do end-to-end storage allocation (optimized) / deallocation / management from a DS81k through VIOs to an AIX LPAR so I know how to do it, but I'm not at liberty to share that code, much less contribute it.

happy If anyone else has thoughts about implementing something, please let me know.

Like (0)

Andreas Trawoeger
109 posts since
Apr 10, 2008

Currently Being Moderated

1. May 13, 2008 8:07 AM (in response to kpg123)
Re: Beta AIX zenpack for Beta Zenoss

"kpg123" wrote:

I've just added a beta version of an AIX Zenpack for the beta version of Zenoss. Perhaps "beta" is too strong of a recommendation for this ZenPack -- all I will guarantee is that it won't end all life on the planet (okay, maybe just Earth is safe... :)

Thanks a lot, I just gave it a try and I can assure you that your AIX Zenpack isn't the primary cause for the zhengzhou earthquake.

I'm having problems with IP Services + CPU performance collection, but I haven't yet asked the AIX guys to change their snmpdv3.conf according to your suggestions.

But at least they didn't came beating me up, like the last time when I spawned 100 telnet sessions on their boxes happy

Report Abuse

Like (0)
kpg123
157 posts since
Jun 29, 2007

Currently Being Moderated

2. May 17, 2008 1:09 PM (in response to Andreas Trawoeger)
Re: Beta AIX zenpack for Beta Zenoss

"atrawog" wrote:

"kpg123" wrote:

I've just added a beta version of an AIX Zenpack for the beta version of Zenoss. Perhaps "beta" is too strong of a recommendation for this ZenPack -- all I will guarantee is that it won't end all life on the planet (okay, maybe just Earth is safe... :)

Thanks a lot, I just gave it a try and I can assure you that your AIX Zenpack isn't the primary cause for the zhengzhou earthquake.

Sigh. No matter how many times you tell people, "Don't touch the big red button!", there's always someone who tries... ;)

"atrawog" wrote:

I'm having problems with IP Services + CPU performance collection, but I haven't yet asked the AIX guys to change their snmpdv3.conf according to your suggestions.

One issue that Zenoss has (at least wrt AIX) is that if when a new AIX server is added, Zenoss sometimes decides that instead of being SNMP v1, that the the AIX client is SNMP v2c. You may want to check that zProperty as well.

Please let me know how it goes. If they have different experiences or suggestions, I'd like to hear it!

"atrawog" wrote:

But at least they didn't came beating me up, like the last time when I spawned 100 telnet sessions on their boxes :-)

Heh. There might be easier ways to meet people... :)

kells

Report Abuse

Like (0)
ftanner
79 posts since
Aug 3, 2007

Currently Being Moderated

3. May 23, 2008 11:45 AM (in response to kpg123)
Re: Beta AIX zenpack for Beta Zenoss

"kpg123" wrote:

I've just added a beta version of an AIX Zenpack for the beta version of Zenoss. Perhaps "beta" is too strong of a recommendation for this ZenPack -- all I will guarantee is that it won't end all life on the planet (okay, maybe just Earth is safe... :)

This is an egg-style zenpack, so it most assuredly won't work with 2.1.3.

http://www.zenoss.com/community/wiki/user-contributed/

**Use this ZenPack only with a test Zenoss system to make sure that there's no possibility of messing up your other performance data.**

First, the bad news...

Known issues:
* requires SNMP version 3 with authentication off (ie SNMP v1 commands will work with this feature set). Description of modifications required in the description for the /devices/servers/AIX class.
* doesn't properly map to the manufacture links for any of the hardware found (eg disk, cpus)
* doesn't allow for re-exports of the ZenPack
* the performance collectors have no error checking and are written in bash and perl (Did I mention the word "beta" yet?)
* the performance collectors explicitly require that version 1 snmpwalk and snmpget work, and won't accept other credential info
* filesystem events don't insert the name of the filesystem
* sanity checks in the modelers seem to want too much sanity, as they *seem* to sometimes prevent modeling (aka discovery) from gathering all information. On the plus side, the sanity checking does prevent Zenoss from blowing away previous modelled device info when it can't talk to the SNMP agents or the machine is down. :)
* the AIXBase modeler still doesn't seem to properly identify the hardware model and update the correct spot for the hardware. The serial number does appear to be updated correctly for the hardware, however.
* doesn't show the per-processor statistics, only a total averaged number. In addition, the MIB doesn't update the statistics all that frequently (I might have missed a setting that can up the polling on CPU usage), so it may even be fantastically wrong.

.. and now the good news:

Features:
* uses standard IBM AIX snmpd services. No net-snmp daemons required, no other packages to install.
* adds subclasses for /devices/servers/AIX, with further subclasses for LPARs and VIO servers. Adds a token link into /devices/server/Linux/HMC with a customized zLink so that it's (perhaps?) easier to get to the HMC console.
* (YMMV) discovers memory, paging space, processors, disks, adapters and filesystems. NB: FRU information and model numbers would (I *think*) require updates to other ZenModel relationships, which would need to be in the base Zenoss.
* has a copy of the AIX MIB (from the samples directory) and cleaned up version of the xmtopas-generated MIB (the MIB it generates is, to be polite, syntactically challenged. It may also be plain wrong in places. :)
* allows for a lot more performance statistics to be gathered

While fixes are especially welcome, other feedback is also appreciated. :)

Thank you,

kells

Future possibilites:
Note that storage-related info such as VG + LV + filesystem + MPIO path + LUN ID mapping will require ZenModel relationships to be created, which will require time. This would be useful to be able to create a report to determine (for example) which filesystems would be affected by an outage of which SAN storage system. Some info may also not be available through SNMP.

The VIO mapping of virtual adapters to physical adapters and LUN assignments might be impossible through SNMP. I have written Perl scripts to do end-to-end storage allocation (optimized) / deallocation / management from a DS81k through VIOs to an AIX LPAR so I know how to do it, but I'm not at liberty to share that code, much less contribute it. happy If anyone else has thoughts about implementing something, please let me know.

I have access to AIX machines. What can I do to help you get this ZenPack "production ready"?

The server that I am running it against currently will only monitor NIC usage and network routes. Nothing else. None of the perf graphs are filling out. No file systems.

I get errors for the /perf/memory and /perf/cpu parts that the commands will not execute.

The lo0 NIC thresholds are erroring out that they are using over 75% of the utilization.

I have "Bad Value Reading" errors for the memAvailReal, xmdMemRealEcomp, laLoadInit5, xmdMemRealEpinned, Real Non-computational, xmdMemRealEfree, and xmdMemRealEconcomp OIDs.

Report Abuse

Like (0)
kpg123
157 posts since
Jun 29, 2007

Currently Being Moderated

4. May 24, 2008 12:20 AM (in response to ftanner)
Re: Beta AIX zenpack for Beta Zenoss

"ftanner" wrote:

I have access to AIX machines. What can I do to help you get this ZenPack "production ready"?

By doing exactly what you're doing -- trying it out and letting me know about issues! Thank you! (Of course, being patient helps too! :)

"ftanner" wrote:

The server that I am running it against currently will only monitor NIC usage and network routes. Nothing else. None of the perf graphs are filling out. No file systems.

Okay, that's not such a good sign -- the NIC usage and network routes are a part of the standard package, and not anything new that I've created. :(

I sometimes see strangeness like that when SNMP commands fail against the AIX server. I'm not sure what you have set up so far, so maybe I can impose on you to give me a little more description of your situation.

For instance:

* What version of AIX are you running?

* Is it possible for you to modify the snmpd files as described in the /Devices/AIX grouping page? (You may have done this already, but I didn't see it in your post)

* What version of Zenoss are you running?

* Can you copy and paste the output of when you run Manage -> Model... menu option?

* Can you check that your AIX host is indeed responding correctly to snmpwalk commands for OIDs like the ones in the templates?

That should give me a good start in determining how I can help you.

As a sanity check, you do see the new MIBs, right? And you do see the organizers in the device tree, right?

"ftanner" wrote:

I get errors for the /perf/memory and /perf/cpu parts that the commands will not execute.

"The commands will not execute"? I'm not sure that I've seen that before. Please post the output and I'll see if I can make sense of it.

"ftanner" wrote:

The lo0 NIC thresholds are erroring out that they are using over 75% of the utilization.

I haven't implemented NIC performance checking yet, so these are the "standard" Zenoss ones.

"ftanner" wrote:

I have "Bad Value Reading" errors for the memAvailReal, xmdMemRealEcomp, laLoadInit5, xmdMemRealEpinned, Real Non-computational, xmdMemRealEfree, and xmdMemRealEconcomp OIDs.

Okay, can I get you to post the exact error messages?

Thank you!

kells

Report Abuse

Like (0)
ftanner
79 posts since
Aug 3, 2007

Currently Being Moderated

5. May 28, 2008 11:20 AM (in response to kpg123)
Re: Beta AIX zenpack for Beta Zenoss

I haven't forgotten about this....

I am working with my AIX guys to get the appropriate information from then.

I will then send you an e-mail in case it's alot of info and/or includes screen caps.

Report Abuse

Like (0)
Andreas Trawoeger
109 posts since
Apr 10, 2008

Currently Being Moderated

6. Jun 3, 2008 7:20 AM (in response to ftanner)
Re: Beta AIX zenpack for Beta Zenoss

Hi kpg123!
"kpg123" wrote:

One issue that Zenoss has (at least wrt AIX) is that if when a new AIX server is added, Zenoss sometimes decides that instead of being SNMP v1, that the the AIX client is SNMP v2c. You may want to check that zProperty as well.

Please let me know how it goes. If they have different experiences or suggestions, I'd like to hear it!

I finally managed to get a new Zenoss 2.2 installation and a AIX 5.3 test host up and running and the SNMP version bug is fixed in Zenoss 2.2.

There is only a slight bug with your Zenpack: The AIX collector scripts aren't included in the pack.

The templates search for them at ${here/ZenPackManager/packs/ZenPacks.community.AIX/path}/libexec/, but the won't find them because they aren't installed by the ZenPack.

cu andreas

Report Abuse

Like (0)
kpg123
157 posts since
Jun 29, 2007

Currently Being Moderated

7. Jun 3, 2008 5:59 PM (in response to Andreas Trawoeger)
Re: Beta AIX zenpack for Beta Zenoss

"atrawog" wrote:

I finally managed to get a new Zenoss 2.2 installation and a AIX 5.3 test host up and running and the SNMP version bug is fixed in Zenoss 2.2.

Oh really? I saw something about that in the release notes for 2.2 but hadn't actually looked at testing it. Good news!

"atrawog" wrote:

There is only a slight bug with your Zenpack: The AIX collector scripts aren't included in the pack.

The templates search for them at {here/ZenPackManager/packs/ZenPacks.community.AIX/path}/libexec/, but the won't find them because they aren't installed by the ZenPack.

cu andreas

I found that out last Friday when Frank pointed that out. sad I repackaged it and also rewrote the collectors in Python, but didn't post the new package. Doh! Thanks for the poke! I've put it onto the wiki page now.

There are still issues with the package and with my test installation. Hopefully I can resolve that soon....

kells

Report Abuse

Like (0)
Andreas Trawoeger
109 posts since
Apr 10, 2008

Currently Being Moderated

8. Jun 4, 2008 6:38 AM (in response to kpg123)
Re: Beta AIX zenpack for Beta Zenoss

"kpg123" wrote:

There are still issues with the package and with my test installation. Hopefully I can resolve that soon....

I have just tried it and it does work pretty well.

The performance templates generate a handful of errors (memAvailReal, xmdMemRealEcomp, xmdMemRealEpinned, Real Non-computational, xmdMemRealEfree, xmdMemRealEnoncomp) and I will look into that.

But my AIX guys will love you for reporting file system mount points like "/" instead of logical volume names like "/dev/hd4".

Report Abuse

Like (0)
kpg123
157 posts since
Jun 29, 2007

Currently Being Moderated

9. Jun 4, 2008 2:04 PM (in response to Andreas Trawoeger)
Re: Beta AIX zenpack for Beta Zenoss

"atrawog" wrote:

I have just tried it and it does work pretty well.

Really? I mean, "Oh good!". :)

"atrawog" wrote:

The performance templates generate a handful of errors (memAvailReal, xmdMemRealEcomp, xmdMemRealEpinned, Real Non-computational, xmdMemRealEfree, xmdMemRealEnoncomp) and I will look into that.

Those all appear to be related to xmtopas (PTX stuff). You may need to run xmtopas in the background, IIRC....

Something like:

xmtopas -C >/dev/null 2>&1 &

(To be honest, I'd have to go look it up to see if you need LPAR security settings in place etc to the underlying LPAR data collection methods so that xmtopas can provide that info. IIRC, there's a setting to prevent playing with performance counters (or some such thing) in order to establish a cover communications channel. I haven't been to any place that actually does Orange Book (or greater) style / Compartmental Mode System style security, so it doesn't apply to me and therefore I don't even remember it.... :)

I'd love to put in a better help page for this stuff, but the /devices/servers/AIX class notes are already obnoxiously long. There has to be a better way to provide that info, but that's what I've got for the moment. Suggestions?

"atrawog" wrote:

But my AIX guys will love you for reporting file system mount points like "/" instead of logical volume names like "/dev/hd4".

Heh. That type of thing was getting to me. :)

Just to check to see what else might be working for you, are the CPU number and type showing up properly? As well as memory and paging space? How about any hdisks?

BTW, thanks for the feedback!

kells

Report Abuse

Like (0)
Andreas Trawoeger
109 posts since
Apr 10, 2008

Currently Being Moderated

10. Jun 5, 2008 8:08 AM (in response to kpg123)
Re: Beta AIX zenpack for Beta Zenoss

"kpg123" wrote:

Those all appear to be related to xmtopas (PTX stuff). You may need to run xmtopas in the background, IIRC....

Something like: xmtopas -C >/dev/null 2>&1 &

What AIX an xmtopas version are you using? On my box xmtopas doesn't have an '-C' option and starts up via inetd:
(scse):root:/wgkk/bin>oslevel -s 5300-06-04-0748 (scse):root:/wgkk/bin>cat /etc/inetd.conf | grep topas xmquery dgram udp wait root /usr/bin/xmtopas xmtopas -p3

"kpg123" wrote:

I'd love to put in a better help page for this stuff, but the /devices/servers/AIX class notes are already obnoxiously long. There has to be a better way to provide that info, but that's what I've got for the moment. Suggestions?

Some kind of wiki could be nice, but for the moment the forum / mailing list will do the job too.
"kpg123" wrote:

"atrawog" wrote:

But my AIX guys will love you for reporting file system mount points like "/" instead of logical volume names like "/dev/hd4".

Heh. That type of thing was getting to me. :)

Just to check to see what else might be working for you, are the CPU number and type showing up properly? As well as memory and paging space? How about any hdisks?

CPUs are funny. My test host is entitled to 1 physical CPU that is split up into 2 logical CPU's and I have 3 CPU's showing up in Zenoss. So either they are added or the host was entitled to 3 logical CPU's somewhere in the past and the info is still available via SNMP. Memory, paging space, hdisks and vpath info shows up correctly.

Another nice thing would be to name interface something like "en1 Gigabit Ethernet-SX Adapter" instead of "en1; Product: Gigabit Ethernet-SX PCI-X Adapter Manufacturer: not available! Part Number: 00P4499 FRU Number: 00P4499". The name is pretty long and and tends to break the Zenoss web layout a little bit?

Thank for all the efforts, you are saving everybody a lot of time :-)

cu andreas

Report Abuse

Like (0)
kpg123
157 posts since
Jun 29, 2007

Currently Being Moderated

11. Jun 9, 2008 2:55 PM (in response to Andreas Trawoeger)
Re: Beta AIX zenpack for Beta Zenoss

"atrawog" wrote:

"kpg123" wrote:

Those all appear to be related to xmtopas (PTX stuff). You may need to run xmtopas in the background, IIRC....

Something like: xmtopas -C >/dev/null 2>&1 &

What AIX an xmtopas version are you using? On my box xmtopas doesn't have an '-C' option and starts up via inetd:

Errmm.... Sorry, I meant *topas* not *xmtopas*. Oops!

nohup /usr/bin/topas -C >/dev/null 2>&1 &

topas communicates with xmtopas (which, as you point out, is run in inetd) in order to do the collection. (I did mention "IIRC", so I'll forgive myself this time. :)

"atrawog" wrote:

"kpg123" wrote:

Just to check to see what else might be working for you, are the CPU number and type showing up properly? As well as memory and paging space? How about any hdisks?

CPUs are funny. My test host is entitled to 1 physical CPU that is split up into 2 logical CPU's and I have 3 CPU's showing up in Zenoss. So either they are added or the host was entitled to 3 logical CPU's somewhere in the past and the info is still available via SNMP.

Hmmm..... As a sanity check, when you do an

lsdev | grep proc

do you only see one processor?

"atrawog" wrote:

Memory, paging space, hdisks and vpath info shows up correctly.

Oh good news!

"atrawog" wrote:

Another nice thing would be to name interface something like "en1 Gigabit Ethernet-SX Adapter" instead of "en1; Product: Gigabit Ethernet-SX PCI-X Adapter Manufacturer: not available! Part Number: 00P4499 FRU Number: 00P4499". The name is pretty long and and tends to break the Zenoss web layout a little bit?

That shouldn't be too hard to do. (Famous last words? happy I'll have a quick look over the next couple of days and check it out.

"atrawog" wrote:

Thank for all the efforts, you are saving everybody a lot of time :-)

I hope it's useful, but every time I talk to the Zenoss team about supporting AIX, they keep telling me that there's no demand from customers for it. I'm doing this in order to (hopefully) help break that vicious cycle of no product <==> no demand by putting together something that works reasonably well to help create that demand.

(NB: I'm hoping that at some point Zenoss will feel comfortable with including this code into Zenoss Core so that it's a easier to maintain. I'll formally submit it to them once I can clean up the Ethernet names and the manufacturer links.)

Are there any Zenoss Enterprise customers out there using AIX/ needing AIX support?

kells

Report Abuse

Like (0)
jbaird
166 posts since
Sep 18, 2007

Currently Being Moderated

12. Jun 9, 2008 6:05 PM (in response to kpg123)
Re: Beta AIX zenpack for Beta Zenoss

I monitor ~60 AIX systems, and I know that other users of Zen are also monitoring. I actually use net-snmp AND AIX's snmpd in order with a custom modeler that I wrote to monitor the systems.

I am currently pulling Serial/HW Model/OS Version, Network interfaces, OS processes, Filesystems, Routes, and performance graphs.

Report Abuse

Like (0)
kpg123
157 posts since
Jun 29, 2007

Currently Being Moderated

13. Jun 10, 2008 12:24 AM (in response to jbaird)
Re: Beta AIX zenpack for Beta Zenoss

"jbaird" wrote:

I monitor ~60 AIX systems, and I know that other users of Zen are also monitoring.

Excellent! Hopefully other people will comment with their AIX environments too!

"jbaird" wrote:

I actually use net-snmp AND AIX's snmpd in order with a custom modeler that I wrote to monitor the systems.

I am currently pulling Serial/HW Model/OS Version, Network interfaces, OS processes, Filesystems, Routes, and performance graphs.

Oh cool! Is there anything that you'd be willing to share?

kells

Report Abuse

Like (0)

jbaird

166 posts since
Sep 18, 2007

Currently Being Moderated

14. Jun 10, 2008 9:29 AM (in response to kpg123)

Re: Beta AIX zenpack for Beta Zenoss

kells,

My process is pretty much explained in http://community.zenoss.com/forums/viewtopic.php?t=2891&highlight=aix.

I basically change AIX's snmpd port to 1610, install net-snmp, and proxy certain MIB's to AIX's snmpd using:


proxy -v 1 -c public 127.0.0.1:1610 .1.3.6.1.2.1.2.2.1
proxy -v 1 -c public 127.0.0.1:1610 1.3.6.1.4.1.2.2.1.1.0
proxy -v 1 -c public 127.0.0.1:1610 1.3.6.1.4.1.2.2.1.1.1.0
# Proxy iftable(IF-MIB: Interface Names & Stats), iptable and ifalias
#proxy -v 1 -c public 127.0.0.1:1610 .1.3.6.1.2.1
#proxy -v 1 -c public 127.0.0.1:1610 .1.3.6.1.2.1.4.20.1
# Proxy RFC1213-MIB (Route Info)
proxy -v 1 -c public 127.0.0.1:1610 .1.3.6.1.2.1.4.21.1
# Proxy SysUptime (netsnmp doesn't work)
proxy -v 1 -c public 127.0.0.1:1610 1.3.6.1.2.1.25.1.1.0
proxy -v 1 -c public 127.0.0.1:1610 .1.3.6.1.2.1.1.3.0
# Proxy the SNMP Descr so OS shows up properly in zenoss parsing
proxy -v 1 -c public 127.0.0.1:1610 .1.3.6.1.2.1.1.1.0
proxy -v 1 -c public 127.0.0.1:1610 .1.3.6.1.2.1.1.1.0

For the modeler, I just have some basic scripts that parse the serial number, etc, and pipe them thru net-snmp:

# Execute scripts to get more information
exec check_ibmserial /Admin/Scripts/snmp/getSerial.ksh
exec check_ibmmodel /Admin/Scripts/snmp/getModel.ksh
exec check_ibmmftg /Admin/Scripts/snmp/getMftg.ksh
exec check_ibmos /Admin/Scripts/snmp/getOS.ksh

Report Abuse
Like (0)

1 2 3 4 Previous Next

Go to original post

May 9, 2008 5:19 PM

Beta AIX zenpack for Beta Zenoss

Actions

More Like This