how do you troubleshoot network performance of a web application

Question

This is a pretty naive question, I guess. I frequently need to figure out what if the problem with a web app or is it simply a networking issue - lost packets or a slow connection in any point between, say, your boss on a crappy wifi connection at the airport and the web server, or the CDN and the web server or MySQL and the webserver, or memcached box... You get the idea. Also, how do you tell if it's the DNS server?

Some of you are missing the point. This is not a question about measuring web application performance. This is a question about measuring network performance as a component of web application performance.

Answer 1

Set up tracing (perhaps togglable by a querystring), showing the amount of time spent in heavy sections of the code, and overall server time. If it server creates the page in .2 seconds and it takes 10 seconds to load, you know it's either the network, or the user's browser.

You can also use the Net tab in Firebug, or timeline view in Fiddler to see individual components coming down and how long they take.

Answer 2

To see if it's the DNS server, you can try to bypass the DNS server. You can do that by adding the relevant host names to your hosts file. If, when you have the addresses in the hosts file, the web server seems to be responding much faster, then investigate the DNS further. If site performance still sucks, the problem is most likely elsewhere.

More advanced; You can also query the DNS directly with nslookup or dig. Make sure you query your web site domain's authoritative DNS servers directly to see their response time, as opposed to the response time of the local DNS server, which may have a cached copy of the entries.

Answer 3

The best approach is to baseline your web performance from sites with known good connectivity. Once you have that established, you can then begin the fairly straightforward process of establishing the cause of latency for your particular client <--> web application issue from the client in question.

My recommendations for baselining latency would be pingdom.com - $10/month gets you rock solid (I've been using them for the last six months - every issue they've reported has been real) latency analysis from multiple locations on the Internet as well as monitoring/notification for website down issues.

Of particular importance to your requirements, is that they establish an audit log of your web sites performance from multiple locations in the United States, as well as internationally. This can prove useful if you have a transient issue and you wish to look back historically.

Being able to provide your boss (who is your target audience in the question) a graph of average latency from twelve different locations provides a strong foundation for identifying the client-side issues (WiFi, DNS, laptop problems) versus an issue with the website's base performance AND connection to the Internet.

DNS server issues, of course, can be immediately remedied by having the client place the appropriate hostname->IP mappings in the local /etc/hosts (or, in the case of Windows, C:\windows\system32\drivers\etc\hosts) file - If the issue disappears when you have the appropriate /etc/hosts file configured, it's a very strong indicator that there is an issue in your resolver chain (local resolver, first DNS server in the chain, or canonical)

Answer 4

Can't hurt to perform actual load testing from multiple locations as well. Run a local load test, and then run one from various locations outside of your data center. The local tests should give you a good baseline of what to expect at LAN/local conditions. Then you can compare that to what you get externally, and you can see what sort of loss you're getting on the networking end. Obviously, the external tests are going to have higher load times, but you have to determine what's acceptable in that realm.

Answer 5

In addition to the other recomendations I like to use Fiddler to see exactly how long individual requests are taking and how much badwidth is used. It runs from the client so it shows me how long each request takes from that persepctive.

Answer 6

Also, try YSlow, a great add-in for Firebug:

http://developer.yahoo.com/yslow/

Answer 7

Seems to me you need to know network connections for each process, the amount of traffic for each socket connection, the response time for each socket and a way to visualize the data.

appfirst.com has worked for us.