1

my site has been slowed down last couple of days ... some image doesn't show up ... i got

408 Request Time-out: Server timeout waiting for the HTTP request from the client 

couple of times which never happened before .

i did a little performance checking , basically top command in ssh

this is the result

http://up.akstube.com/images/vdtebe9sr10si1eyycd.jpg

as you can see memory usage is very high ... or i think it is .

i've rebooted the server ... it wend down but after while it goes up again .... (this is about an hour after reboot )

http://up.akstube.com/images/49xjeylxy0st7vwn4ojl.png

after half an hour =>

    top - 16:14:22 up  1:56,  2 users,  load average: 0.62, 0.90, 1.15
    Tasks: 303 total,   1 running, 302 sleeping,   0 stopped,   0 zombie
    Cpu(s):  8.0%us,  0.2%sy,  0.0%ni, 91.4%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
    Mem:  32840004k total, 21037648k used, 11802356k free,   430832k buffers
    Swap:  1050616k total,        0k used,  1050616k free, 16527108k cached

right now

[root@ns4008353 ~]# top
top - 16:33:25 up  2:15,  2 users,  load average: 0.88, 0.94, 0.98
Tasks: 303 total,   3 running, 300 sleeping,   0 stopped,   0 zombie
Cpu(s):  6.8%us,  0.2%sy,  0.0%ni, 92.8%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  32840004k total, 22388160k used, 10451844k free,   434964k buffers
Swap:  1050616k total,        0k used,  1050616k free, 17324104k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7532 mysql     20   0 3750m 492m 5556 S 18.0  1.5  49:53.58 mysqld
12903 apache    20   0  232m  40m 4864 S 16.3  0.1   0:56.74 httpd
 7748 apache    20   0  235m  43m 5256 S 11.6  0.1   1:33.36 httpd
 8010 apache    20   0  262m  70m 4856 R  6.7  0.2   1:32.05 httpd
 7747 apache    20   0  235m  43m 5220 S  1.3  0.1   1:06.37 httpd
 7749 apache    20   0  222m  30m 5164 S  1.3  0.1   1:58.53 httpd
 7996 apache    20   0  250m  59m 5476 S  1.3  0.2   1:37.37 httpd
   10 root      20   0     0    0    0 S  0.3  0.0   0:09.72 rcu_sched
 7714 apache    20   0  265m  73m 4972 S  0.3  0.2   1:34.53 httpd
 7905 named     20   0  669m  24m 3020 S  0.3  0.1   0:02.01 named
 7999 apache    20   0  232m  40m 4968 S  0.3  0.1   1:17.67 httpd
20865 root      20   0     0    0    0 S  0.3  0.0   0:01.05 kworker/2:2
21491 root      20   0 15212 1308  852 S  0.3  0.0   0:01.38 top
23810 root      20   0 15212 1340  852 R  0.3  0.0   0:00.07 top
    1 root      20   0 19404 1568 1268 S  0.0  0.0   0:00.58 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd

free m

[root@ns4008353 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         32070      21897      10173          0        424      16955
-/+ buffers/cache:       4516      27553
Swap:         1025          0       1025

do you think this the reason of slow response ? is there any comment to get more info about what is using the memory ?


i have a dedicate server :

Processor Name  Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
Vendor ID   GenuineIntel
Processor Speed (MHz)   1600.000
Total Memory    32840004 kB
Free Memory 10416752 kB
Total Swap Memory   1050616 kB
Free Swap Memory    1050616 kB
System Uptime   0 Days, 2 Hours and 22 Minutes
Apache 2.2.24   Running
DirectAdmin 1.43.0  Running
Exim 4.76   Running
MySQL 5.5.31    Running
Named 9.8.2rc1  Running
ProFTPd 1.3.4d  Running
sshd    Running
dovecot 2.2.4   Running

Php 5.3.26 Installed

after looking in my log files i've seen lots of this error

[Sat Sep 28 00:15:41 2013] [error] [client 66.249.73.162] Request exceeded the limit of 10 internal redirects due to probable co
nfiguration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace
5
  • Any other information of interest, like has your site gotten a lot more popular, or an uptick in HTTP requests per minute? Or any new code changes on the site? Are the images that are timing out hosted locally (hard disk) or externally (NFS mount or third party website)? Sep 28, 2013 at 15:50
  • Also do some local requests for webpages (from the server) and from some various different sites other than your work or office to compare. Could be your provider is messing up routing or traffic intermittently, or your home ISP just really sucks. If a local request never times out, that's the case. If not, then you have server/code issues. Sep 28, 2013 at 16:04
  • 'some image doesn't show up', is this a test you did or someone else said happened? What is your client timeout set to on the server? Sep 28, 2013 at 17:36
  • @inetplumber , no it has about 100-150 thousand visits per day . no changes in the code , images are hosted internally
    – max
    Sep 28, 2013 at 19:11
  • just viewed your edit, so now grep 66.249.73.162 in your transfer log and see what page/code the request triggered. Could be a bug or a maliciously crafted request that's not being protected against (also a bug I guess). Sep 29, 2013 at 23:06

3 Answers 3

2

it's usually .htaccess file that re-injecting a changed request URI , check that out and also you can reset your config files

0

Your problem description is slow HTTP requests but, as you showed, CPU and memory are not exhausted. So a couple of things to check:

  • Make sure that the requests really are slow. In httpd.conf, find this line:

LogFormat "%h %l %u %t \"%r\" %>s %b" common

And add %D after %b (but inside the ending quote). See http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats

Then reload/restart httpd and your access_log file should now show the time each response takes in microseconds in the last column. Confirm these are slow. Maybe only some types of requests are slow? I notice that MySQL is using some CPU, so perhaps one type of request is hitting a slow database query?

  • If the responses above really are slow and there is no obvious pattern (e.g. particular pages), then you need to determine where they're slow. The %D time includes everything, from reading the request from the client, serving the file (which includes any application code or database queries), and writing the response, so it just tells you there is a slowdown, not necessarily where. So now you need to isolate where the slow down is. I would first check the application. Also check the MySQL slow log (enable it if it isn't enabled).

  • Check your error_log - maybe there are some application timeouts or other errors.

  • If you're not having any luck, you can also run pstack ${PID} periodically to try to understand what httpd is doing.

  • Finally, you can use tcpdump and something like Wireshark to isolate where the slowdown is, although that will be a bit tedious. I think there is also an httpd module that breaks down the time in a request, but I can't find it now and don't see it in the list of built-in modules.

2
  • thanks , why do every one says the memory usage is not high ? haven't you seen the first image ? it consumes the whole 32g couple hours after reboot
    – max
    Sep 28, 2013 at 19:13
  • As @sandeep.s85 mentions, your memory usage is not high. As he also mentions, you can look on the -/+ buffers line in free -m, or in top, you can add the cached amount to free.
    – Infofinity
    Sep 29, 2013 at 5:25
0

Your memory usage is not high, you should skip that part from troubleshooting and try what Inforfinity has suggested.

The Linux kernel tends to occupy the available unused memory for disk caching. It makes the system faster. If any application need memory, it will take that from the cache itself. So basically you are not low on memory. Consider the output of free -m

-/+ buffers/cache: 4516 27553

4516MB of Actual RAM is being used, 27553 is the actual free memory. Close to 17G is cached.

Try to do performance testing on your web server. Use httperf to benchmark.

httperf --server yoursite.com --port 80 --num-conns 100 --rate 10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .