How can I determine very specifically what is causing my web server to get slower?

Question

I suspect that some particular large file or URL is being hit a lot, but I am failing to sort out which one it is. Are there some standard troubleshooting tools or steps recommended? Thanks!

Answer 1

Well my first instinct, if I believed that the slowdown was due to excessive traffic on a URL or file, would be to check the web server logs or any analytics you might have. This will tell you what traffic you are experiencing.

Answer 2

A few things you can check if you assume it is a bandwidth related issue:

• A mentioned in another answer check access/error logs to get a history of any clients creating more traffic than normal. If it is a small site you might be able to manually look at the logs. Otherwise look into a log parsing/visualization for that type of log.
• If your server provider has a network usage graph/display for your server you can use that as a rough guide to see if/when there are any spikes in usage.
• You can use ''ifconfig -a'' to look at the raw usage of your server's network interfaces.
• You can use ''netstat -an' to list all the current connections.
• Use the status module of Apache/lighttpd (or a similar module for your server application) to display the current list of connected clients.
• Don't assume the issue is necessarily bandwidth related, especially if nothing stands out as abnormal from the logs/stats.
• Consider installing some sort of monitoring application in case you find it difficult to find the issue when it occurs.

The following a two scripts I use to get a basic idea of the status of connections on a server using netstat. The first one just displays the number of connections based on the connection type:

  #!/bin/sh
  # Display number of connections on a server
  #
  echo -n $"Established: "
  netstat -an | grep ESTABLISHED | wc -l

  echo -n $"   Syn Recv: "
  netstat -an | grep SYN | wc -l

  echo -n $"       Wait: "
  netstat -an | grep WAIT | wc -l

  echo -n $"     Listen: "
  netstat -an | grep LISTEN | wc -l

  echo -n $"      Total: "
  netstat -an | wc -l

Note that is is normal to have a large number of connections in the WAIT state (at least from what I've observed). A large number of SYN connections may be to do a SYN flood DOS attack.

The following script counts the number of established connections by IP address and is useful to see if there is one particular misbehaving IP address.

#!/bin/sh
# Counts the number of connections by IP address

netstat -an | grep ESTABLISHED | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

If your server is under heavy load and you have 1000s of connections this script can take a little bit to run (~1 minute in worst case I've seen).

Note that depending on your application multiple entries per IP is not necessarily bad. For example, in my case 10-20 entries/IP is normal but beyond that amount is suspicious. I've seen 'bad' IP addresses that either request the same document/file 100s of times or walk through every file on the site and download it. The last script makes it relatively easy to spot these IPs which you can then use to look in your logs for more details if needed.

Answer 3

There is no general way of debugging poor webserver performance, but there are some tools which should be helpful for you. I assume you're using some http stack under linux.

1. Log the amout of time needed to process each request. (in apache you can set LogFormat to: LogFormat %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T/%D combined ane then try to determine the longest GETs or POSTs from the logfile.
2. Almost realtime logfile analysis can be done using apachetop command (for logfiles in apache format, not necessarily generated by apache) apachetop -f /var/log/apache2/access.log will give you good outlook on pending requests
3. On the client side you can use some webdeveloper's tools to determine which elements of the busiest site are loading to slowly. I'd recommnend firebug http://getfirebug.com/ plugin for firefox.
4. Check disk performance of the server. Using atop http://www.atoptool.nl/ you can quickly determine which disks are 100% busy. This can be problem especially when you're using database engine (mysql or pgsql) on the same hosts and experience high traffic.
5. Big static files shouldn't be served by apache. The best way of getting apache working well, is to serve everything you can, by other, lightweight servers - (i.e. nginx, lighthttpd...). Try to find the bigest file served by your webserver and serve them through nginx.
6. Check also bandwith usage statistics. If youre reaching the your network limits, pages will get load much slower and server will get high load.

Every sysadmin who have fighted such problems as your, would probably enumerate many more quick hints, but the fact is, that all this rules depends very much on the environment (hardware, operatoring system, webserver, application, database) and can be only learned by experience. Often quite painfull.

Answer 4

Consider monitoring the system with a tool like sar or Munin. This will give you information on which component are busiest. I like Munin for its graphical reporting, also the ability to set alerts. sar gives good details on disk and swap activity.

Including the service time in the Apache log is useful. I usually change the remote logname (ident) field (%l) with the time taken to serve the request (%T) in the extended log format.