Archived community.zenoss.org | full text search
Skip navigation
Up to Discussions in zenoss-users
4377 Views 9 Replies Latest reply: Sep 9, 2011 10:28 AM by Shane Scott RSS
moss2k10 Rank: White Belt
moss2k10
24 posts since
Dec 13, 2010
Currently Being Moderated

Sep 1, 2011 6:10 AM

Snmp agent down on ASA - new problem?

Hello,

 

I'm having SNMP agent down problem - zMaxOIDPerRequest decreased to 20 or even 10 did not solve it.

 

I'm able to successfully snmpwalk (version 3) the device from zenoss server .

 

What is more running  zenperfsnmp run -v 10 -d devicename also works.

 

SNMP agent down remains and no OS/graph's data is being updated.

 

Restarting zenoss/zenperfsnmp/readding the device checked and doesn't help.

 

Zenperfsnmp logs from both (working and snmp_agent_down) devices attached.

 

 

Any ideas?

 

Peter

Attachments:
Tags: snmp, cisco, issues, asa
  • Shane Scott ZenossMaster
    Shane Scott
    1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    1. Sep 1, 2011 10:53 AM (in response to moss2k10)
    Re: Snmp agent down on ASA - new problem?

    moss2k10:

     

    I'd reccomend verifying that theres no packet loss between the zenoss colelctor and the problem device.

     

    Next I'd suggest moving it to another colelctor and back to ensure the config is pushed:

    `su zenoss`

    `zendmd`

    d=dmd.Devices.findDevice('problem device')

    d.setPerformanceMonitor('localhost-temp')

    commit()

    sync()

    d.pushconfig()

    commit()

    sync()

    d.setPerformanceMonitor('localhost')

    commit()

    sync()

    d.pushConfig()

     

    If you're using distributed collectors be sure to substitute 'localhost' with the correct collector name. Additionally be sure to update the host collector of the device post operation. Do note that this will restart all the daemons on that collector.

     

    Let me know if it works. If not we can try some other items.

     

    Best,

    --Hackman238

  • moss2k10 Rank: White Belt
    moss2k10
    24 posts since
    Dec 13, 2010
    Currently Being Moderated
    2. Sep 2, 2011 4:46 AM (in response to Shane Scott)
    Re: Snmp agent down on ASA - new problem?

    What I've noticed yesterday is that problem doesn't exist when I add affected devices to other zenoss servers.

    The version which we use is 2.5.2 - on one server all is ok and on the other one snmp is down for several devices.

    I've noticed something more on tcpdump:

     

    Correct zenoss - device communication:

     

    15:00:27.485613 IP zenoss.38807 > device.snmp:  F=r U= E=  C= GetRequest(14) [|snmp]

    15:00:27.486308 IP device.snmp > zenoss.38807:  [msgFlags=0x08]

    15:00:27.486477 IP zenoss.38807 > device.snmp:  F=apr [|snmp][|snmp]

    15:00:27.488003 IP device.snmp > zenoss.38807:  F=ap [|snmp][|snmp]

    and so on...

     

    Incorrect zenoss - device communication:

     

    15:01:05.406032 IP zenoss.39639 > device.snmp:  F=r U= E=  C= GetRequest(14) [|snmp]

    15:01:05.433021 IP device.snmp > zenoss.39639:  [msgFlags=0x08]

     

    In second case zenoss server doesn't send 3rd message and I don't know why..

    Another thing is DNS entries - nonworking devices have different IP's being resolved under hostnames binded in zenoss - can this be a cause of such problems?

     

    I forgot to mention - almost all of them are ASA active-standby failover pairs

  • Shane Scott ZenossMaster
    Shane Scott
    1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    3. Sep 2, 2011 8:48 AM (in response to moss2k10)
    Re: Snmp agent down on ASA - new problem?

    moss2k10:

     

    I'm not familliar with the ASA failover pairing. Are you monitoring the devices via a VIP or are you hitting intentionally assigned management interfaces on each?

     

    --Hackman238

  • moss2k10 Rank: White Belt
    moss2k10
    24 posts since
    Dec 13, 2010
    Currently Being Moderated
    4. Sep 2, 2011 9:15 AM (in response to Shane Scott)
    Re: Snmp agent down on ASA - new problem?

    We are monitoring them via standard interfaces used for network traffic.

    It makes no difference cuz they are working on other zenoss installation so they can be monitored in pairs.

    I will reinstall affected zenoss and restore from last backup - we will see if that will help..

  • Shane Scott ZenossMaster
    Shane Scott
    1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    5. Sep 2, 2011 9:56 AM (in response to moss2k10)
    Re: Snmp agent down on ASA - new problem?

    moss2k10:

     

    Pairing complicates monitoring with some devices. If you've it working in another instance the same way then that definately sounds like a local instance problem. Let me know if the restore produces differnt results.

     

    Best,

    --Hackman238

  • moss2k10 Rank: White Belt
    moss2k10
    24 posts since
    Dec 13, 2010
    Currently Being Moderated
    6. Sep 7, 2011 3:47 AM (in response to Shane Scott)
    Re: Snmp agent down on ASA - new problem?

    It was DNS related problem Scott I don't know the details but after changing DNS server,reboot and reinstall zenoss for same version 2.5.2 all devices are working fine ..

     

    Only thing that didn't work is my backup which I can't use because problems are showing up again after restoring..

    So currently I'm playing with zendisc

     

    Thanks for Your time Scott

    Peter

  • moss2k10 Rank: White Belt
    moss2k10
    24 posts since
    Dec 13, 2010
    Currently Being Moderated
    7. Sep 7, 2011 6:33 AM (in response to moss2k10)
    Re: Snmp agent down on ASA - new problem?

    It's amazing ... When I added first 3-5 devices to /Devices/Network/Cisco class all of them worked correctly but when I  added new 40 devices and move firewalls to /Devices/Network/Firewall SNMP agent down error shows again ..

    From log zenperfsnmp I've also noticed some warnings/errors:

     

    Last successful before running zendisc:

     

     

    2011-09-07 09:18:58,103 INFO zen.zenperfsnmp: success:0 fail:0 pending:0 todo:5

    2011-09-07 09:18:58,286 INFO zen.zenperfsnmp: success:1 fail:0 pending:4 todo:0

    2011-09-07 09:18:58,383 INFO zen.zenperfsnmp: success:2 fail:0 pending:3 todo:0

    2011-09-07 09:18:58,410 INFO zen.zenperfsnmp: success:3 fail:0 pending:2 todo:0

    2011-09-07 09:18:58,521 INFO zen.zenperfsnmp: success:4 fail:0 pending:1 todo:0

    2011-09-07 09:18:58,642 INFO zen.zenperfsnmp: ******** Cycle completed ********

    2011-09-07 09:18:58,642 INFO zen.zenperfsnmp: Sent 520 OID requests

    2011-09-07 09:18:58,643 INFO zen.zenperfsnmp: Queried 5 devices

    2011-09-07 09:18:58,643 INFO zen.zenperfsnmp:   0 in queue still unqueried

    2011-09-07 09:18:58,643 INFO zen.zenperfsnmp:   Successes: 5  Failures: 0  Not reporting: 0

    2011-09-07 09:18:58,643 INFO zen.zenperfsnmp: Waited on 0 queries from previous cycles.

    2011-09-07 09:18:58,643 INFO zen.zenperfsnmp:   Successes: 0  Failures: 0  Not reporting: 0

    2011-09-07 09:18:58,643 INFO zen.zenperfsnmp: Cycle lasted 0.54 seconds

    2011-09-07 09:18:58,644 INFO zen.zenperfsnmp: *********************************

    2011-09-07 09:18:58,645 INFO zen.zenperfsnmp: success:5 fail:0 pending:0 todo:0

     

    Then I started zendisc from cli and zenperfsnmp restarted at 10:03 after finishing the discovery (is it normall?):

     

    2011-09-07 10:03:09,102 INFO zen.zenperfsnmp: Initializing daemon...

    2011-09-07 10:03:09,104 INFO zen.zenperfsnmp: Connecting to localhost:8789

    2011-09-07 10:03:09,117 INFO zen.zenperfsnmp: Connected to ZenHub

    2011-09-07 10:03:09,121 INFO zen.zenperfsnmp: Fetching property items...

    2011-09-07 10:03:09,153 INFO zen.zenperfsnmp: Getting threshold classes...

    2011-09-07 10:03:09,158 INFO zen.zenperfsnmp: Checking for outdated configs...

    2011-09-07 10:03:09,176 INFO zen.zenperfsnmp: Fetching configs for ['TEPLWROFW014', 'TEPLWROFW019', 'TEPLWROFW005', 'TEPLWROFW006', 'TEPLWROFW007', 'TEPLWROFW001', 'TEPLWROFW011', 'TEPLWROFW010', 'TEPLWROSW001', 'TEPLWROFW015', 'TEPLWROFW003', 'TEPLWROUPS13', 'us000450.eu.tieto.com', 'TEPLWROUPS05', 'TEPLWROFW020', 'TEPLWROUPS06']...

    2011-09-07 10:03:09,192 INFO zen.zenperfsnmp: Fetching default RRDCreateCommand...

    2011-09-07 10:03:09,194 INFO zen.zenperfsnmp: Getting collector thresholds...

    2011-09-07 10:03:09,209 INFO zen.zenperfsnmp: Fetching SNMP status...

    2011-09-07 10:03:09,214 INFO zen.zenperfsnmp: Initiating incremental device load

    2011-09-07 10:03:13,930 INFO zen.zenperfsnmp: success:0 fail:0 pending:0 todo:0

    2011-09-07 10:03:13,931 INFO zen.zenperfsnmp: ******** Cycle completed ********

    2011-09-07 10:03:13,931 INFO zen.zenperfsnmp: Sent 0 OID requests

    2011-09-07 10:03:13,931 INFO zen.zenperfsnmp: Queried 0 devices

    2011-09-07 10:03:13,931 INFO zen.zenperfsnmp:   0 in queue still unqueried

    2011-09-07 10:03:13,932 INFO zen.zenperfsnmp:   Successes: 0  Failures: 0  Not reporting: 0

    2011-09-07 10:03:13,932 INFO zen.zenperfsnmp: Waited on 0 queries from previous cycles.

    2011-09-07 10:03:13,932 INFO zen.zenperfsnmp:   Successes: 0  Failures: 0  Not reporting: 0

    2011-09-07 10:03:13,932 INFO zen.zenperfsnmp: Cycle lasted 0.00 seconds

    2011-09-07 10:03:13,932 INFO zen.zenperfsnmp: *********************************

    2011-09-07 10:03:13,963 INFO zen.zenperfsnmp: Configured 16 of 16 devices

     

    And now errors/warnings started to show up:

     

     

    2011-09-07 10:08:09,227 INFO zen.zenperfsnmp: success:0 fail:0 pending:0 todo:22

    2011-09-07 10:08:09,568 ERROR netsnmp: snmp_send: Too long

    2011-09-07 10:08:09,658 ERROR netsnmp: snmp_send: Too long

    2011-09-07 10:08:09,660 ERROR netsnmp: snmp_send: Too long

    2011-09-07 10:08:09,660 WARNING zen.zenperfsnmp: Failed to collect on TEPLWROFW014 (twisted.python.failure.Failure: [Failure instance: Traceback (failure with no frames): pynetsnmp.netsnmp.SnmpError: snmp_send])

     

    2011-09-07 10:08:09,705 INFO zen.zenperfsnmp: success:0 fail:1 pending:7 todo:14

    2011-09-07 10:08:09,705 WARNING zen.zenperfsnmp: TEPLWROFW014 SNMP agent down

     

    This situation is repeating for all devices affected but still I've few that works fine ...

     

    Is it possible that zenperfsnmp isn't able to work with devices which have long names?

    I'm asking because DNS resolves some of these devices with domain name extension making it long e.g.: TEPLSZCSW001.eu.company.com

     

    Thanks

    Peter

  • moss2k10 Rank: White Belt
    moss2k10
    24 posts since
    Dec 13, 2010
    Currently Being Moderated
    8. Sep 8, 2011 4:24 AM (in response to moss2k10)
    Re: Snmp agent down on ASA - new problem?

    Some final results of my tests about zenoss 2.5.x failover ASA pairs monitoring:

    1)after adding active ASA device all works fine - zenperfsnmp shows no errors/warnings

    2)after adding standby ASA device zenperfsnmp loses communication with active ASA - new standby ASA works ok

     

    Configuration details:

    Zenoss     zenoss-2.5.2.el5.x86_64.rpm & zenoss-2.5.1.el5.x86_64.rpm

    Centos     5.6 (Final)

     

    Solution:

    We've added only active devices with exporting syslog to zenoss server to check/alarm failover status

     

    BR

    Peter

  • Shane Scott ZenossMaster
    Shane Scott
    1,373 posts since
    Jul 6, 2009
    Currently Being Moderated
    9. Sep 9, 2011 10:28 AM (in response to moss2k10)
    Re: Snmp agent down on ASA - new problem?

    Pete:

     

    Interesting. Gladd you got it working!

     

    Best,

    --Shane

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points