2

Background

I have a new hosted, dedicated database server behind a dedicated firewall (Cisco ASA 5505 Sec+). The plan is to have a virtual (aka "cloud") web server or two on the other side of the firewall connecting back to the backend DB server.

While setting up the server, I was unimpressed by it's network performance. Turns out, though the 2 servers have GigE - the firewall only supports 100Mb - so most of the performance issues I had can be adequately explained by that.

Problem

However, as part of troubleshooting, I ran a series of pings to the firewall from the dedicated server. These pings came back with some interesting results - specifically, the distribution of 100 pings was:

57% < 1ms
14% between 1ms and 2ms
12% between 2ms and 3ms
11% between 3ms and 6ms
6% >= 6ms
Min/Avg/Max: 0/1/8 ms

I would've expected first hop to be < 1ms consistently (and can't honestly recall any hard wired environment where it wasn't). Subsequent tests were pretty similar, and have been for so me number of days - so this doesn't appear to be an isolated incident. No retransmits or dropped packets have been observed. Pinging across the firewall shows similar performance:

58% < 1ms
14% between 1ms and 2ms
8% between 2ms and 2ms
14% between 3ms and 6ms
6% >= 6ms
Min/Avg/Max: 0/2/56 ms

Troubleshooting

The hoster has checked the server, firewall and intervening switch(es) and sees no issues. They also point out that they "deprioritize" ICMP traffic on the network. They noticed some recent port flapping (likely caused, I believe, by configuration of the server) and will "continue to monitor" the situation. The port flapping isn't numerous enough or time correlated enough to explain the ping times, though it may be possible it's a(nother) symptom of an underlying issue.

I don't have direct access to the ASA - but the hoster ran some stats on it as part of troubleshooting:

# ping ***** (series of 5-packet pings from firewall to server, edited for brevity)
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/10 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/8/10 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/6/10 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

# show cpu usage
CPU utilization for 5 seconds = 13%; 1 minute: 11%; 5 minutes: 10%

# show mem
Free memory:       341383104 bytes (64%)
Used memory:       195487808 bytes (36%)
-------------     ----------------
Total memory:      536870912 bytes (100%)

# show int eth0/1
Interface Ethernet0/1 "", is up, line protocol is up
Hardware is 88E6095, BW 100 Mbps, DLY 100 usec
    Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)
    Available but not configured via nameif
    MAC address *****, MTU not set
    IP address unassigned
    5068644 packets input, 5077178693 bytes, 0 no buffer
    Received 4390 broadcasts, 0 runts, 0 giants
    0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
    0 L2 decode drops
    387883 switch ingress policy drops
    3220647 packets output, 1648213382 bytes, 0 underruns
    0 output errors, 0 collisions, 0 interface resets
    0 babbles, 0 late collisions, 0 deferred
    0 lost carrier, 0 no carrier
    0 input reset drops, 0 output reset drops
    0 rate limit drops
    0 switch egress policy drops

Except for some seemingly high CPU usage for a firewall with a few ACLs and only possibly an RDP session going through it, I don't see anything alarming about the ASA stats. It certainly doesn't appear over-taxed IMHO.

Question

Considering we're approaching disk seek times, and there's no production traffic on the firewall or server yet - I'm still a little concerned. What do you guys think? Is this an issue? Is this normal in a larger data-center environment?

4
  • 1
    What specific ASA model are we talking about? 5500+ doesn't really tell us much.
    – pauska
    Sep 3, 2013 at 13:26
  • I don't think I would have bothered with the ASA in this scenario at all; the endpoint firewalls on the two servers would be more than sufficient for anything short of a DDoS, and the ASA wouldn't handle that either. Sep 3, 2013 at 13:27
  • @MichaelHampton - FWIW, the ASA also serves VPN duties and will be used for site-to-site VPN (not currently setup, so I don't think it's affecting this latency issue) in the near-future. That, and I still am not entirely comfortable with host-based firewalls.... Sep 3, 2013 at 14:55
  • And I'm not exactly comfortable with firewalls I don't have access to. :) Sep 3, 2013 at 14:59

1 Answer 1

2

First off, you're not saying what specific ASA model you have, nor the licensing mode. Please post the output of "sh ver" and "sh int Ethernet0/0".

That being said - different ASA models have different throughput limits. As an example the ASA5510 has a maximum throughput (concurrent) limit of 300mbps. See http://www.cisco.com/en/US/prod/collateral/vpndevc/ps6032/ps6094/ps6120/product_data_sheet0900aecd802930c5.html for the complete list.

When it comes to latency - all Cisco products puts traffic directly against the device in the bottom queue. This is why it's bad practice to ICMP echo against a router or a firewall, as the results are never predictable. We have two ASA5510's here (both with gigabit), and two 3750-X switches, and they all jump up to 300ms ICMP echo latency when they are pushing a lot of traffic.

This does not mean that routed/forwarded traffic is slow

If you want to check latency then use ping between devices across the ASA. It's the only reliable way.

6
  • It's a 5505 Security+. I'm not sure on licensing, but it's a 100Mb port and max throughput seems consistent with that. It also has client-to-site and site-to-site VPN licensed. It's notated as "Port Upgrade: 100Mb Port" on the account - though I don't really see how that'd be an "upgrade". I've added details of pings through the ASA to the question as well - which show similar behavior. Again, this is a new setup with no other devices and no actual production traffic short of an RDP session to the server. Sep 3, 2013 at 14:52
  • @MarkBrackett are there any routers between the firewall and the server(s)? How many switches are involved in the data flow?
    – pauska
    Sep 3, 2013 at 15:03
  • Just got this update from the hoster: "the firewall is on one switch while the server is on another with a router in between. The router has the VLAN's trunked across it to allow communication between the switches...the extra hops could be the cause of the latency". Sep 3, 2013 at 15:05
  • FWIW, though, I'm really trying to get an answer as to whether I should be concerned about this or not, not necessarily the cause of the additional latency if it's pretty much SOP in a hosted environment. It seems abnormally high to me, but I'm not used to a hosted environment (though the hoster is top-tier with the premium prices to match) or large-scale networking - so it's possible this is just the norm and I'm being a PITA.... Sep 3, 2013 at 15:10
  • @MarkBrackett the thumb rule is that each router adds 2ms of processing delay when doing routing. This however sounds like someone using a router as a switch (why god, whyyyy). I have no idea how routers affect switched packets, but I presume that this is the cause of your troubles. Maybe you should ask them to put the firewall on the same switch as your server(s).
    – pauska
    Sep 3, 2013 at 15:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .