Sep 1, 2011 6:10 AM
Snmp agent down on ASA - new problem?
-
Like (0)
Hello,
I'm having SNMP agent down problem - zMaxOIDPerRequest decreased to 20 or even 10 did not solve it.
I'm able to successfully snmpwalk (version 3) the device from zenoss server .
What is more running zenperfsnmp run -v 10 -d devicename also works.
SNMP agent down remains and no OS/graph's data is being updated.
Restarting zenoss/zenperfsnmp/readding the device checked and doesn't help.
Zenperfsnmp logs from both (working and snmp_agent_down) devices attached.
Any ideas?
Peter
moss2k10:
I'd reccomend verifying that theres no packet loss between the zenoss colelctor and the problem device.
Next I'd suggest moving it to another colelctor and back to ensure the config is pushed:
`su zenoss`
`zendmd`
d=dmd.Devices.findDevice('problem device')
d.setPerformanceMonitor('localhost-temp')
commit()
sync()
d.pushconfig()
commit()
sync()
d.setPerformanceMonitor('localhost')
commit()
sync()
d.pushConfig()
If you're using distributed collectors be sure to substitute 'localhost' with the correct collector name. Additionally be sure to update the host collector of the device post operation. Do note that this will restart all the daemons on that collector.
Let me know if it works. If not we can try some other items.
Best,
--Hackman238
What I've noticed yesterday is that problem doesn't exist when I add affected devices to other zenoss servers.
The version which we use is 2.5.2 - on one server all is ok and on the other one snmp is down for several devices.
I've noticed something more on tcpdump:
Correct zenoss - device communication:
15:00:27.485613 IP zenoss.38807 > device.snmp: F=r U= E= C= GetRequest(14) [|snmp]
15:00:27.486308 IP device.snmp > zenoss.38807: [msgFlags=0x08]
15:00:27.486477 IP zenoss.38807 > device.snmp: F=apr [|snmp][|snmp]
15:00:27.488003 IP device.snmp > zenoss.38807: F=ap [|snmp][|snmp]
and so on...
Incorrect zenoss - device communication:
15:01:05.406032 IP zenoss.39639 > device.snmp: F=r U= E= C= GetRequest(14) [|snmp]
15:01:05.433021 IP device.snmp > zenoss.39639: [msgFlags=0x08]
In second case zenoss server doesn't send 3rd message and I don't know why..
Another thing is DNS entries - nonworking devices have different IP's being resolved under hostnames binded in zenoss - can this be a cause of such problems?
I forgot to mention - almost all of them are ASA active-standby failover pairs
moss2k10:
I'm not familliar with the ASA failover pairing. Are you monitoring the devices via a VIP or are you hitting intentionally assigned management interfaces on each?
--Hackman238
We are monitoring them via standard interfaces used for network traffic.
It makes no difference cuz they are working on other zenoss installation so they can be monitored in pairs.
I will reinstall affected zenoss and restore from last backup - we will see if that will help..
moss2k10:
Pairing complicates monitoring with some devices. If you've it working in another instance the same way then that definately sounds like a local instance problem. Let me know if the restore produces differnt results.
Best,
--Hackman238
It was DNS related problem Scott I don't know the details but after changing DNS server,reboot and reinstall zenoss for same version 2.5.2 all devices are working fine ..
Only thing that didn't work is my backup which I can't use because problems are showing up again after restoring..
So currently I'm playing with zendisc
Thanks for Your time Scott
Peter
It's amazing ... When I added first 3-5 devices to /Devices/Network/Cisco class all of them worked correctly but when I added new 40 devices and move firewalls to /Devices/Network/Firewall SNMP agent down error shows again ..
From log zenperfsnmp I've also noticed some warnings/errors:
Last successful before running zendisc:
2011-09-07 09:18:58,103 INFO zen.zenperfsnmp: success:0 fail:0 pending:0 todo:5
2011-09-07 09:18:58,286 INFO zen.zenperfsnmp: success:1 fail:0 pending:4 todo:0
2011-09-07 09:18:58,383 INFO zen.zenperfsnmp: success:2 fail:0 pending:3 todo:0
2011-09-07 09:18:58,410 INFO zen.zenperfsnmp: success:3 fail:0 pending:2 todo:0
2011-09-07 09:18:58,521 INFO zen.zenperfsnmp: success:4 fail:0 pending:1 todo:0
2011-09-07 09:18:58,642 INFO zen.zenperfsnmp: ******** Cycle completed ********
2011-09-07 09:18:58,642 INFO zen.zenperfsnmp: Sent 520 OID requests
2011-09-07 09:18:58,643 INFO zen.zenperfsnmp: Queried 5 devices
2011-09-07 09:18:58,643 INFO zen.zenperfsnmp: 0 in queue still unqueried
2011-09-07 09:18:58,643 INFO zen.zenperfsnmp: Successes: 5 Failures: 0 Not reporting: 0
2011-09-07 09:18:58,643 INFO zen.zenperfsnmp: Waited on 0 queries from previous cycles.
2011-09-07 09:18:58,643 INFO zen.zenperfsnmp: Successes: 0 Failures: 0 Not reporting: 0
2011-09-07 09:18:58,643 INFO zen.zenperfsnmp: Cycle lasted 0.54 seconds
2011-09-07 09:18:58,644 INFO zen.zenperfsnmp: *********************************
2011-09-07 09:18:58,645 INFO zen.zenperfsnmp: success:5 fail:0 pending:0 todo:0
Then I started zendisc from cli and zenperfsnmp restarted at 10:03 after finishing the discovery (is it normall?):
2011-09-07 10:03:09,102 INFO zen.zenperfsnmp: Initializing daemon...
2011-09-07 10:03:09,104 INFO zen.zenperfsnmp: Connecting to localhost:8789
2011-09-07 10:03:09,117 INFO zen.zenperfsnmp: Connected to ZenHub
2011-09-07 10:03:09,121 INFO zen.zenperfsnmp: Fetching property items...
2011-09-07 10:03:09,153 INFO zen.zenperfsnmp: Getting threshold classes...
2011-09-07 10:03:09,158 INFO zen.zenperfsnmp: Checking for outdated configs...
2011-09-07 10:03:09,176 INFO zen.zenperfsnmp: Fetching configs for ['TEPLWROFW014', 'TEPLWROFW019', 'TEPLWROFW005', 'TEPLWROFW006', 'TEPLWROFW007', 'TEPLWROFW001', 'TEPLWROFW011', 'TEPLWROFW010', 'TEPLWROSW001', 'TEPLWROFW015', 'TEPLWROFW003', 'TEPLWROUPS13', 'us000450.eu.tieto.com', 'TEPLWROUPS05', 'TEPLWROFW020', 'TEPLWROUPS06']...
2011-09-07 10:03:09,192 INFO zen.zenperfsnmp: Fetching default RRDCreateCommand...
2011-09-07 10:03:09,194 INFO zen.zenperfsnmp: Getting collector thresholds...
2011-09-07 10:03:09,209 INFO zen.zenperfsnmp: Fetching SNMP status...
2011-09-07 10:03:09,214 INFO zen.zenperfsnmp: Initiating incremental device load
2011-09-07 10:03:13,930 INFO zen.zenperfsnmp: success:0 fail:0 pending:0 todo:0
2011-09-07 10:03:13,931 INFO zen.zenperfsnmp: ******** Cycle completed ********
2011-09-07 10:03:13,931 INFO zen.zenperfsnmp: Sent 0 OID requests
2011-09-07 10:03:13,931 INFO zen.zenperfsnmp: Queried 0 devices
2011-09-07 10:03:13,931 INFO zen.zenperfsnmp: 0 in queue still unqueried
2011-09-07 10:03:13,932 INFO zen.zenperfsnmp: Successes: 0 Failures: 0 Not reporting: 0
2011-09-07 10:03:13,932 INFO zen.zenperfsnmp: Waited on 0 queries from previous cycles.
2011-09-07 10:03:13,932 INFO zen.zenperfsnmp: Successes: 0 Failures: 0 Not reporting: 0
2011-09-07 10:03:13,932 INFO zen.zenperfsnmp: Cycle lasted 0.00 seconds
2011-09-07 10:03:13,932 INFO zen.zenperfsnmp: *********************************
2011-09-07 10:03:13,963 INFO zen.zenperfsnmp: Configured 16 of 16 devices
And now errors/warnings started to show up:
2011-09-07 10:08:09,227 INFO zen.zenperfsnmp: success:0 fail:0 pending:0 todo:22
2011-09-07 10:08:09,568 ERROR netsnmp: snmp_send: Too long
2011-09-07 10:08:09,658 ERROR netsnmp: snmp_send: Too long
2011-09-07 10:08:09,660 ERROR netsnmp: snmp_send: Too long
2011-09-07 10:08:09,660 WARNING zen.zenperfsnmp: Failed to collect on TEPLWROFW014 (twisted.python.failure.Failure: [Failure instance: Traceback (failure with no frames): pynetsnmp.netsnmp.SnmpError: snmp_send])
2011-09-07 10:08:09,705 INFO zen.zenperfsnmp: success:0 fail:1 pending:7 todo:14
2011-09-07 10:08:09,705 WARNING zen.zenperfsnmp: TEPLWROFW014 SNMP agent down
This situation is repeating for all devices affected but still I've few that works fine ...
Is it possible that zenperfsnmp isn't able to work with devices which have long names?
I'm asking because DNS resolves some of these devices with domain name extension making it long e.g.: TEPLSZCSW001.eu.company.com
Thanks
Peter
Some final results of my tests about zenoss 2.5.x failover ASA pairs monitoring:
1)after adding active ASA device all works fine - zenperfsnmp shows no errors/warnings
2)after adding standby ASA device zenperfsnmp loses communication with active ASA - new standby ASA works ok
Configuration details:
Zenoss zenoss-2.5.2.el5.x86_64.rpm & zenoss-2.5.1.el5.x86_64.rpm
Centos 5.6 (Final)
Solution:
We've added only active devices with exporting syslog to zenoss server to check/alarm failover status
BR
Peter
Pete:
Interesting. Gladd you got it working!
Best,
--Shane
Follow Us On Twitter »
|
Latest from the Zenoss Blog » | Community | Products | Services Resources | Customers Partners | About Us | ||
Copyright © 2005-2011 Zenoss, Inc.
|
||||||||