I've been trying to get to the bottom of this for a while now and, well, it seems like I can't. We have two servers behind an ASA5505 (software version 8.3) at a datacenter. They run a wide variety of services, including our website, internal XMPP server, game servers (Minecraft and Team Fortress 2, both using UDP for the most part), mail...
Every day roughly around noon PST the network speed becomes absolutely terrible for about an hour while the system load of the firewall goes from the usual 30% to above 80%. According to show processes cpu-hog
, "Quack process" (what the duck?!) and especially "Dispatch Unit" are, well, hogging the CPU a bit.
There seems to be a pattern when the network goes bad. For about 2 seconds it's at full speed, then it slows down to almost a halt for 2 more. I've enabled logging to ssh during this, and nothing interesting showed up. Just a few blocked ICMP requests and, a bit odd, Deny IP due to Land Attack from [one of our IPs] to [the exact same IP]
, but that might be an actual attack?
Anyways, speed is bad from and to the two servers and also to the firewall itself, making me thing that it's overburdened, although the ping between the two servers is always good. I'm not sure how the network is set up exactly though, so there might simply be a small switch between the firewall and the servers.
Another weird thing, but, again, this might be normal (couldn't find anything about it), in show threat-detection statistics
the internal IPs of our servers/VMs show up first and some actually have numbers greater than 0 for fw-drop
.
What should I try next time this issue comes around? Any ideas as to what might cause this? Should I disable limit-policy-map (see below)?
EDIT: Pinging the servers from the firewall will also show these symptoms.
Here's some more system information:
access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list outside_in; 33 elements; name hash: 0xc5896c24
access-list outside_in line 1 extended permit tcp any object-group www_servers object-group www_srv 0x9c6770f3
access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq ftp (hitcnt=2443) 0x73b87a74
access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq ssh (hitcnt=27915) 0x73a19ab3
access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq www (hitcnt=21568957) 0x045edf43
access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq https (hitcnt=19746) 0xe54a2315
access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 3389 (hitcnt=3919) 0x58629d3c
access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 30 (hitcnt=134) 0xcd3db679
access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 5922 (hitcnt=43) 0x17c6f16b
access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 6122 (hitcnt=1) 0x3ea3c2e6
access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 2200 (hitcnt=2) 0x8356fbc6
access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq 5722 (hitcnt=1) 0xaefada3e
access-list outside_in line 1 extended permit tcp any(65536) object-group www_servers(1) eq domain (hitcnt=17) 0x45c7e0b1
access-list outside_in line 2 extended permit udp any object-group www_servers object-group www_srv_udp 0x9426d24f
access-list outside_in line 2 extended permit udp any(65536) object-group www_servers(1) eq 3389 (hitcnt=1) 0x15cdc545
access-list outside_in line 2 extended permit udp any(65536) object-group www_servers(1) eq domain (hitcnt=4468079) 0x1b6d6b19
access-list outside_in line 3 extended permit icmp host [...] any (hitcnt=0) 0x155d597f
access-list outside_in line 4 extended permit icmp host [...] any (hitcnt=289) 0x0fcc844a
access-list outside_in line 5 extended permit icmp any object-group www_servers echo-reply 0x46f79e30
access-list outside_in line 5 extended permit icmp any(65536) object-group www_servers(1) echo-reply (hitcnt=97) 0x53984766
access-list outside_in line 6 extended permit tcp host [...] eq 25565 host 10.5.209.12 eq 25565 (hitcnt=0) 0x60c828e6
access-list outside_in line 7 extended permit tcp any object-group mc eq 25565 0xcb0d2f17
access-list outside_in line 7 extended permit tcp any(65536) object-group mc(6) eq 25565 (hitcnt=478488) 0x3ce89b9a
access-list outside_in line 8 extended permit tcp any object-group irc object-group ircd 0x65619a8f
access-list outside_in line 8 extended permit tcp any(65536) object-group irc(8) eq 6667 (hitcnt=6336) 0xda23eb42
access-list outside_in line 8 extended permit tcp any(65536) object-group irc(8) eq 6969 (hitcnt=8445981) 0xb39f9de5
access-list outside_in line 9 extended permit tcp any object-group rob object-group xmppd 0x24db3318
access-list outside_in line 9 extended permit tcp any(65536) object-group rob(9) eq 5222 (hitcnt=2836) 0x3b220aef
access-list outside_in line 9 extended permit tcp any(65536) object-group rob(9) eq 5269 (hitcnt=316) 0x8c4a1677
access-list outside_in line 10 extended permit udp any object-group rob object-group xmppd 0x56997935
access-list outside_in line 10 extended permit udp any(65536) object-group rob(9) eq 5222 (hitcnt=0) 0x1378a09e
access-list outside_in line 10 extended permit udp any(65536) object-group rob(9) eq 5269 (hitcnt=0) 0x484e999c
access-list outside_in line 11 extended permit udp any object-group tf2_servers object-group tf2_udp_ports 0x4ed88dd7
access-list outside_in line 11 extended permit udp any(65536) object-group tf2_servers(12) range 26901 27009 (hitcnt=20) 0x984f0cfd
access-list outside_in line 11 extended permit udp any(65536) object-group tf2_servers(12) range 27015 27024 (hitcnt=1842395) 0x5117dbf3
access-list outside_in line 12 extended permit tcp any object-group tf2_servers object-group tf2_tcp_ports 0xd792e8d1
access-list outside_in line 12 extended permit tcp any(65536) object-group tf2_servers(12) eq 8080 (hitcnt=16028) 0x1f9dcdd6
access-list outside_in line 13 extended permit object-group tcp_udp any object-group rob object-group mumble_ports 0x62e8f226
access-list outside_in line 13 extended permit tcp any(65536) object-group rob(9) eq 64738 (hitcnt=4) 0x663e2204
access-list outside_in line 13 extended permit udp any(65536) object-group rob(9) eq 64738 (hitcnt=14) 0x3751c05a
access-list outside_in line 14 extended permit udp any object-group kfy_servers object-group kfy_ports 0x928ebaab
access-list outside_in line 14 extended permit udp any(65536) object-group kfy_servers(16) eq 9009 (hitcnt=52) 0x3c77464e
access-list outside_in line 15 extended permit udp any host 10.5.209.10 object-group bittorrent 0x20a28a30
access-list outside_in line 15 extended permit udp any host 10.5.209.10(168153354) eq 10299 (hitcnt=44693845) 0x140f0e51
access-list outside_in line 16 extended permit tcp any host 10.5.209.10 object-group bittorrent 0xfe939491
access-list outside_in line 16 extended permit tcp any host 10.5.209.10(168153354) eq 10299 (hitcnt=3763575) 0x1ef0e366
access-list outside_in line 17 extended permit icmp any object-group rob 0x6f990c22
access-list outside_in line 17 extended permit icmp any(65536) object-group rob(9) (hitcnt=1418) 0x8401a397
access-list limiter; 3 elements; name hash: 0x189b5c6d
access-list limiter line 1 extended deny ip host [...] any (hitcnt=0) 0x72cb4f57
access-list limiter line 2 extended deny ip host 10.0.0.0 any (hitcnt=0) 0x3d376866
access-list limiter line 3 extended permit ip any any (hitcnt=89047566) 0x1bc67ee2
policy-map limit-policy-map
class limit-map
set connection per-client-max 500 per-client-embryonic-max 30
set connection timeout embryonic 0:00:10 half-closed 0:05:00 dcd
policy-map global_policy
class inspection_default
inspect dns
inspect ftp
class-map limit-map
match access-list limiter
class-map inspection_default
match default-inspection-traffic
class-map limit