Got a head-scratcher here. Either I've found a Windows and/or VMware bug, or I've missed something stupendously simple. [Spoiler - it was stupendously simple.]
Our VMware environment is ESXI 5.5. We have a Windows 2012 R2 VM (named BOS-NETMON) that can no longer do SNMP queries of Cisco ASA devices (v2 or v3).
This machine runs SolarWinds Orion, and neither that nor standalone tools like Paessler SNMPTEST work. All other currently-monitored devices continue to respond to SNMP from this guest - we have Cisco IOS, Meraki, Exagrid, APC/Schneider - all fine. All other allowed protocols (SSH, HTTPS, ICMP) work when connecting to the ASAs from this guest.
The problem started when we switched the guest from the E1000e vNIC hardware to the VMXNet3. Prior to that, it worked fine for a couple of years.
- Running the ASDM Logging window, it doesn't even see the attempted SNMP connection from BOS-NETMON. EDIT - This was because I didn't have the right logging settings on the ASA.
- Running Wireshark on BOS-NETMON, it shows the SNMP queries going out, doesn't register responses from the ASAs.
- I've tested SNMP from another VMXNet3 guest, it also fails to query SNMP against an ASA but works against non-ASA Cisco devices. EDIT - this doesn't appear to be true. Either I tested wrong, or it started working recently. Either way, another host with a VMXNet3 NIC can query SNMP against one of these ASAs.
- I've tested SNMP from a guest with the E1000e vNIC, and it successfully queries SNMP against an ASA.
- 4/5 of the target ASAs are not at the same site, so it shouldn't be an ARP issue - and if it were, the other non-SNMP protocols would be affected.
Further edit: I have ASDM debug info for a successful, and an unsuccessful, SNMP session. Next step will be a packet capture, I guess.
6 Mar 12 2018 16:30:26 302015 10.50.100.177 63809 10.10.99.10 161 Built inbound UDP connection 610885144 for Inside_Interface:10.50.100.177/63809 (10.50.100.177/63809) to identity:10.10.99.10/161 (10.10.99.10/161) 7 Mar 12 2018 16:30:26 710005 10.50.100.152 49588 10.10.99.10 161 UDP request discarded from 10.50.100.152/49588 to Inside_Interface:10.10.99.10/161