I have a new hosted, dedicated database server behind a dedicated firewall (Cisco ASA 5505 Sec+). The plan is to have a virtual (aka "cloud") web server or two on the other side of the firewall connecting back to the backend DB server.
While setting up the server, I was unimpressed by it's network performance. Turns out, though the 2 servers have GigE - the firewall only supports 100Mb - so most of the performance issues I had can be adequately explained by that.
However, as part of troubleshooting, I ran a series of pings to the firewall from the dedicated server. These pings came back with some interesting results - specifically, the distribution of 100 pings was:
57% < 1ms 14% between 1ms and 2ms 12% between 2ms and 3ms 11% between 3ms and 6ms 6% >= 6ms Min/Avg/Max: 0/1/8 ms
I would've expected first hop to be < 1ms consistently (and can't honestly recall any hard wired environment where it wasn't). Subsequent tests were pretty similar, and have been for so me number of days - so this doesn't appear to be an isolated incident. No retransmits or dropped packets have been observed. Pinging across the firewall shows similar performance:
58% < 1ms 14% between 1ms and 2ms 8% between 2ms and 2ms 14% between 3ms and 6ms 6% >= 6ms Min/Avg/Max: 0/2/56 ms
The hoster has checked the server, firewall and intervening switch(es) and sees no issues. They also point out that they "deprioritize" ICMP traffic on the network. They noticed some recent port flapping (likely caused, I believe, by configuration of the server) and will "continue to monitor" the situation. The port flapping isn't numerous enough or time correlated enough to explain the ping times, though it may be possible it's a(nother) symptom of an underlying issue.
I don't have direct access to the ASA - but the hoster ran some stats on it as part of troubleshooting:
# ping ***** (series of 5-packet pings from firewall to server, edited for brevity) Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/10 ms Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms Success rate is 100 percent (5/5), round-trip min/avg/max = 1/8/10 ms Success rate is 100 percent (5/5), round-trip min/avg/max = 1/6/10 ms Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms # show cpu usage CPU utilization for 5 seconds = 13%; 1 minute: 11%; 5 minutes: 10% # show mem Free memory: 341383104 bytes (64%) Used memory: 195487808 bytes (36%) ------------- ---------------- Total memory: 536870912 bytes (100%) # show int eth0/1 Interface Ethernet0/1 "", is up, line protocol is up Hardware is 88E6095, BW 100 Mbps, DLY 100 usec Full-Duplex(Full-duplex), 100 Mbps(100 Mbps) Available but not configured via nameif MAC address *****, MTU not set IP address unassigned 5068644 packets input, 5077178693 bytes, 0 no buffer Received 4390 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 L2 decode drops 387883 switch ingress policy drops 3220647 packets output, 1648213382 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collisions, 0 deferred 0 lost carrier, 0 no carrier 0 input reset drops, 0 output reset drops 0 rate limit drops 0 switch egress policy drops
Except for some seemingly high CPU usage for a firewall with a few ACLs and only possibly an RDP session going through it, I don't see anything alarming about the ASA stats. It certainly doesn't appear over-taxed IMHO.
Considering we're approaching disk seek times, and there's no production traffic on the firewall or server yet - I'm still a little concerned. What do you guys think? Is this an issue? Is this normal in a larger data-center environment?