ZenPacks : IBM Websphere MQ Components : Comments

RE: IBM Websphere MQ Components

community@zenoss.org — Wed, 21 Nov 2012 18:46:09 GMT

Hi Joseph,

We really like the look of this but we are having to tweak the scripts about somewhat. Wondered if you have further experience or feedback from anyone else??

We are using MQ 7.0.1.5 and have some queues that include the string RQMNAME so, in Common.py processing queues we ran foul of your test for QMNAME. This meant we ended up with a Manager called ENABLED for many queues. I think we may have similar problems with Channels because we have some called FAIL.

The good news for us is that we can set the zCommandUser and password to values that can directly login to the mq systems and run the dspmq and runmqsc commands directly, without needing to

su so, in your modelers we have replaced commands like:

/opt/zenoss/ZenPacks/ZenPacks.community.zenIbmMQ-1.1-py2.7.egg/ZenPacks/community/zenIbmMQ/modeler/plugins/MqQueueMapUnix.py

Change line from:

"echo 'DISPLAY QUEUE(*) ALL' | su - mqm -c \"runmqsc $MGR\" ; done"

To:

"echo 'DISPLAY QUEUE(*) ALL' | runmqsc $MGR ; done"

For some reason, we also had problems with:

/opt/zenoss/ZenPacks/ZenPacks.community.zenIbmMQ-1.1-py2.7.egg/ZenPacks/community/zenIbmMQ/modeler/plugins/Common.py

Change line from:

if device.zMqRunsOnUnix == True:

To:

if device.zMqRunsOnUnix:

... it didn't pass the True test otherwise???

Any other feedback gratefully received.

Cheers,

Jane

RE: IBM Websphere MQ Components

community@zenoss.org — Mon, 26 Nov 2012 15:51:04 GMT

Hi Jane,

The issues you mentioned ring a bell and I think i remember running into them myself and dealing with them "on the fly" so to speak, and looks like I was lax in publishing the changes.

A while back I also found an issue with the device/component relations (preventing the zenchkrels script from finishing), a mistake on my part that unfortunately affects almost all of my zenpacks (basically i used a plural name from dev->comp, and a singular name from comp->rel, when they should both be the same), so I've been having to go back and fix that in a bunch of them. While doing so, I decided I need to go ahead and make them 4.x compatible, so i have to make the recommended changes there as well...

The point of all this is that I think it will take a while before i can make all of the needed updates (and publish them), but it's on my plate for the near future (since I need to complete them before I can upgrade to 4.x myself).

Since i don't have anything "clean" to present here, i'll paste the text of my current "Common.py" for you to compare against what you have. I remember dealing with the issues you mentioned (the "ENABLED" QM name and the wierd "True" bug, so I think they should be OK in this file.

Regarding the "su - mqm -c", i believe that was necessary for our environment (b/c the "mqm" user had the right ENV vars/permissions). I'm definitely no MQ expert, and may have incorrectly assumed that this was the general case. Maybe I should put that command string in a zProperty rather than hard-coding it?

Anyways I hope to have something more "polished" to post soon, but in the meantime hopefully this will help:

--------------------Common.py:

import re,os

from Products.ZenUtils.Utils import prepId

from Products.DataCollector.plugins.CollectorPlugin import CollectorPlugin

from Products.DataCollector.plugins.DataMaps import ObjectMap

from ZenPacks.community.zenIbmMQ.MQHandler import MQParser

from subprocess import *

class MQComponentMap():

"""

deviceProperties = (

'zMqRunsOnUnix',

)

def findUnixBundleKey(self,input):

""" correct a bug with the class inheritance

"""

for line in input.split('\n'):

if re.search('CHANNEL',line) != None:

self.bundleKey = 'CHANNEL'

break

if re.search('QUEUE',line) != None:

self.bundleKey = 'QUEUE'

break

if re.search('QMNAME',line) != None:

self.bundleKey = 'QMNAME'

break

def process(self, device, results, log):

""" adds Zenoss components based on given list of dictionaries

"""

log.debug('Collecting MQ components for device %s with results %s' % (device.id,results))

common = Common()

dataDict = {}

if device.zMqRunsOnUnix == True:

self.findUnixBundleKey(results)

dataDict = common.unixDictionary(results)

else:

self.bundleKey,output = results

dataDict = common.windowsDictionary(output,device.id,device.zWinUser,device.zWinPassword,self.bundleKey)

objects = common.parseResults(self.bundleKey,dataDict)

rm = self.relMap()

log.debug('Found %s components of type %s' %(len(objects),self.bundleKey))

for o in objects:

om = self.objectMap(o)

rm.append(om)

return rm

class Common():

"""

def __init__(self):

"""

self.parser = MQParser()

self.wincommon = WindowsCommon()

def ManagerInfo(self,dataDict):

""" Map MQ data to component attributes

"""

info = {}

name = dataDict['QMNAME']

info['id'] = prepId(name)

info['managerName'] = name

#info['managerStatus'] = dataDict['STATUS']

return info

def QueueInfo(self,manager,dataDict):

""" Map MQ data to component attributes

"""

info = {}

name = dataDict['QUEUE']+'_'+manager

name = re.sub('[^A-Za-z0-9]+', '', name)

info['id'] = prepId(name)

info['queueName'] = dataDict['QUEUE']

info['queueManager'] = manager

info['queueType'] = dataDict['TYPE']

info['queueMaxDepth'] = dataDict['MAXDEPTH']

return info

def ChannelInfo(self,manager,dataDict):

""" Map MQ data to component attributes

"""

info = {}

name = dataDict['CHANNEL']+'_'+manager+'_'+dataDict['CONNAME']+'_'+dataDict['CHLTYPE']

name = re.sub('[^A-Za-z0-9]+', '_', name)

info['id'] = prepId(name)

info['channelName'] = dataDict['CHANNEL']

info['channelConn'] = dataDict['CONNAME']

info['channelType'] = dataDict['CHLTYPE']

info['channelStatus'] = dataDict['STATUS']

info['channelManager'] = manager

return info

def parseBundle(self,manager,bundles,bundleKey):

""" return dictionary objects for components

"""

output = []

for bundle in bundles:

self.parser.lines = bundle

self.parser.info = {}

self.parser.dataMap()

try:

test = self.parser.info[bundleKey]

if bundleKey == 'CHANNEL':

if re.search('SYSTEM',test) == None:

output.append(self.ChannelInfo(manager,self.parser.info))

if bundleKey == 'QUEUE':

ignoreFlag = False

if re.search('SYSTEM',test) != None:

ignoreFlag = True

if re.search('COM.IBM.MQ.PCF',test) != None:

ignoreFlag = True

if ignoreFlag == False:

output.append(self.QueueInfo(manager,self.parser.info))

if bundleKey == 'QMNAME':

output.append(self.ManagerInfo(self.parser.info))

except:

pass

return output

def unixDictionary(self,input):

"""

dataDict = {}

resultlines = input.split("\n")

manager = ''

addLine=False

for line in resultlines:

if re.search("QueueManager ",line) != None: # new manager

manager = line[len("QueueManager "):len(line)]

dataDict[manager] = []

addLine=True

elif re.search("^QMNAME",line) != None: # parsing output of dspmq

manager = line[line.find("(")+1:line.find(")")]

dataDict[manager] = []

line = re.sub(' +',' ',line)

addLine=True

if addLine == True:

dataDict[manager].append(line)

return dataDict

def windowsDictionary(self,input,device,user,password,bundleKey):

"""

dataDict = {}

self.parser.lines = input

for line in self.parser.cleanLines():

key,value = self.parser.keyValue(line)

if key == 'QMNAME': # name of queue manager

manager = value

dataDict[manager] = self.wincommon.getWindowsComponents(device,user,password,manager,bundleKey)

return dataDict

def parseResults(self,bundleKey,dataDict):

"""

beginString = "QueueManager "

endString = "All valid MQSC commands were processed"

midString = ''

objects = []

if bundleKey == 'QMNAME':

for manager,list in dataDict.items():

objects = objects + self.parseBundle(manager,[list],bundleKey)

return objects

else:

midString = "BLAHBLAH"

if bundleKey == 'CHANNEL':

midString = "AMQ8417"

elif bundleKey == 'QUEUE':

midString = "AMQ8409"

for manager,list in dataDict.items():

bundles = self.parser.bundleData(list,beginString,midString,endString)

objects = objects + self.parseBundle(manager,bundles,bundleKey)

return objects

class WindowsCommon():

"""

def __init__(self):

"""

self.winexe = '/opt/zenoss/bin/winexe'

self.newline = '\r\n'

def connectInit(self,device,user,password):

self.authString = user+'%'+password

self.destString = '//' + device

def getManagers(self):

""" retrieve list of queue managers

"""

localCommand = 'echo | /opt/zenoss/bin/winexe -U "'+self.authString+'" '+self.destString+' "dspmq"'

output = Popen([localCommand],stdout=PIPE,shell=True)

return output.stdout.readlines()

def getQueues(self,manager):

""" retrieve list of mq channel data

"""

localCommand = '/usr/bin/printf "DISPLAY QUEUE(*) ALL\nEND\n" |'+\

'/opt/zenoss/bin/winexe -U "'+self.authString+'" '+self.destString+\

' "runmqsc '+manager+'"'

output = Popen([localCommand],stdout=PIPE,shell=True)

return output.stdout.readlines()

def getChannels(self,manager):

""" retrieve list of mq channel data

"""

localCommand = '/usr/bin/printf "DISPLAY CHSTATUS(*) ALL\nEND\n" |'+\

'/opt/zenoss/bin/winexe -U "'+self.authString+'" '+self.destString+\

' "runmqsc '+manager+'"'

output = Popen([localCommand],stdout=PIPE,shell=True)

return output.stdout.readlines()

def getWindowsComponents(self,device,user,password,manager,bundleKey):

"""

output = []

self.connectInit(device, user, password)

if re.search('CHANNEL',bundleKey) != None:

return self.getChannels(manager)

elif re.search('QUEUE',bundleKey) != None:

return self.getQueues(manager)

elif re.search('QMNAME',bundleKey) != None:

return self.getManagers()

--------------------end

Regards,

Joseph

RE: IBM Websphere MQ Components

community@zenoss.org — Wed, 13 Feb 2013 21:00:07 GMT

Hello Joseph,

Your zenpack is awesome, however on RHEL4 it lists only "MQ Queues" components for me, missing MQ Channel and MQ Manager. Any ideas on troubleshooting technique?

PS: We are using Zenoss v.4.1.1

RE: IBM Websphere MQ Components

community@zenoss.org — Fri, 15 Feb 2013 16:27:11 GMT

Hello,

Glad to hear you like the zenpack, hope it's useful.

I haven't yet tested it myself against the 4.x versions of Zenoss (but will soon), but assuming that the version is not the issue, there are a couple of things you can do to test.

1) run the modeler from the command line, like:

zenmodeler run -v10 -d DEVICENAME --collect=MqQueueMapUnix

and examine the output (or post it here)

2) look for the specific commands being run within the modeler files and try running them manually from the command line. The zenpack uses SSH to run the commands, assumes that "mqm" is the user that MQ runs as (probably should pull that out into a zProperty), and the zCommandUser must have permissions to run the following (for the queues):

3) try this from the RHEL CLI (bash script but may have syntax errors so YMMV):

for LINE in $(dspmq|awk '{print $1}'); do MGR=$(echo $LINE | cut -f 2 -d '(' | cut -f 1 -d ')'); printf "QueueManager ${MGR}\n"; echo 'DISPLAY QUEUE(*) ALL' | su - mqm -c "runmqsc $MGR" ; done

4) the Queue Manager command is :

su - mqm -c dspmq

Hope this helps,

Joseph

RE: IBM Websphere MQ Components

community@zenoss.org — Fri, 15 Feb 2013 23:11:01 GMT

Hello Joseph,

I re-ran all three components from Zenoss server via CLI:

zenmodeler run -v10 -d <MQServer> --collect=MqQueueMapUnix
zenmodeler run -v10 -d <MQServer> --collect=MqManagerMapUnix
zenmodeler run -v10 -d <MQServer> --collect=MqChannelMapUnix

All three seem to work properly and return the expected results, however the only info I see under Zenoss GUI Components is related to queues, Channels and Managers folders are missing altogether.

PS: I can email you screenshots if you want to eyeball it.

Thank you so much for quick turnaround

RE: IBM Websphere MQ Components

community@zenoss.org — Thu, 21 Feb 2013 15:53:13 GMT

Hi Joseph

I'd like to use your ZenPack but I'm having trouble when installing it in my Zenoss 4.2.3 server. I have posted the errors below. I know you said you have not tested it on v4 yet, but any ideas how I can get it working?

Thanks in advance

Graham

[zenoss@spin-ldn-montest tmp]$ zenpack --install ZenPacks.community.zenIbmMQ-1.1-py2.6.egg.zip

ERROR:zen.ZenPackCmd:zenpack command failed

Traceback (most recent call last):

File "/opt/zenoss/Products/ZenUtils/zenpack.py", line 486, in <module>

zp.run()

File "/opt/zenoss/Products/ZenUtils/zenpack.py", line 182, in run

os.makedirs(skinsSubdir, 0750)

File "/opt/zenoss/lib/python2.7/os.py", line 150, in makedirs

makedirs(head, mode)

File "/opt/zenoss/lib/python2.7/os.py", line 157, in makedirs

mkdir(name, mode)

OSError: [Errno 20] Not a directory: '/opt/zenoss/Products/ZenPacks.community.zenIbmMQ-1.1-py2.6.egg/skins'

RE: IBM Websphere MQ Components

community@zenoss.org — Tue, 02 Apr 2013 14:19:58 GMT

I've released the 2.0 version of this Zenpack, which should be Zenoss 4.x compatible.

Joseph