Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Image downloading from our site slows down every evening. Compared to daytime we get %50 more traffic over the evening. In exact figures, during daytime hours, hourly pageview count is around 20k, during evening hours hourly pageview count is around 30k.

We are on Apache, Centos 5, PHP 5, MySQL.

The graph below shows a specific 3MB image's download time from a specific remote site every 10 minutes. As site traffic increases, download speed drops to half.

alt text

  • Our cpu usage is not high. We have two quad-core cpus.
  • 32GB memory.
  • Mysql uses about 10GB of that memory.
  • Apache server-status shows around 40-50 concurrent processes during peak times. Each process holding about 30-40MB resident memory. So roughly 2GB memory goes to Apache.

The rest of the memory, around 20GB, is left for linux to use. vmstat, free -m do not show any swapping. Here is a vmstat output:

# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0    220 409700 491656 25727136    0    0     9    42    1    1  1  0 98  0  0
 2  0    220 408076 491656 25727140    0    0     0  2032 1916 1992  2  1 98  0  0
 0  0    220 405848 491656 25727180    0    0     0     0 1631 1325  1  0 99  0  0
 0  0    220 404992 491656 25727180    0    0     0     0 1508 1203  1  0 99  0  0
 3  0    220 405228 491656 25727216    0    0     0     0 1548 1600  2  1 98  0  0
 0  0    220 404236 491656 25727224    0    0     0     0 1860 1645  2  1 97  0  0
 1  0    220 403916 491656 25727284    0    0     0  2512 1817 1035  2  1 97  0  0
 0  0    220 403900 491656 25727284    0    0     0  2164 1826 1678  1  1 98  0  0
 0  0    220 408908 491656 25727312    0    0     0   828 1636  952  1  0 99  0  0
 0  0    220 408544 491656 25727312    0    0     0     0 1844 1991  1  1 98  0  0
 1  0    220 407856 491664 25727328    0    0   180     0 1567 1194  4  0 96  0  0
 0  0    220 406772 491664 25727460    0    0     0     0 1290 1049  4  1 95  0  0
 0  0    220 406964 491664 25727548    0    0    28   208 1589  904  1  0 99  0  0
 0  0    220 406840 491664 25727560    0    0     0  1796 1885 1396  1  0 99  0  0
 1  0    220 405136 491664 25727612    0    0     0     0 1727 1280  1  0 98  0  0
 0  0    220 404400 491664 25727628    0    0    24     0 1807 1494  1  0 98  0  0
 1  2    220 403996 491668 25727812    0    0   232     0 2221 1633  1  1 97  1  0
 0  0    220 404228 491668 25727844    0    0    20  1776 1673 1332  1  0 97  1  0
 0  0    220 403688 491672 25727988    0    0    68   348 1508  977  1  0 99  0  0
 0  0    220 403444 491672 25728004    0    0     0     0 1436  900  1  0 99  0  0
 0  0    220 403948 491672 25728104    0    0     0     0 1413 1131  1  0 99  0  0
 1  0    220 392984 491672 25728104    0    0     0  1520 1455 1946  5  2 93  0  0
 1  0    220 307264 491672 25728124    0    0     0    12 1829 1520  8  2 91  0  0
 4  0    220 306276 491672 25728132    0    0     0     0 1364 1497  7  0 92  0  0
 1  0    220 305320 491672 25728160    0    0     0     0 1510 1773  9  1 90  0  0
 1  0    220 305444 491672 25728164    0    0     0     0 1620 1666  7  0 93  0  0
 1  0    220 304456 491672 25728208    0    0     0  3396 1813 1836  7  0 93  0  0

Since this specific test image is called every 10 minutes it should stay in the OS Cache and be delivered from the OS Cache, right? The cpu is not busy at all. So the image should be delivered pretty quickly?

What is the bottlenecek I'm hitting here?
-Would it help to install a second harddrive just to host the images?
-How about using PHP APC to reduce CPU load?

Any advices to diagnose the problem?

Thanks, Haluk

UPDATE:
Per Knitti's advice I've setup the same image in a memory mounted directory.
Below are the results for yesterday:
Memory mounted 3MB image:
alt text

Harddrive 3MB image:

alt text

On average they are very close. They are both affected from the evening traffic. The image from the memory can sometimes be delivered slower compared to the harddrive. I think this tells me they are both in the memory as they should be.

So I'm thinking this is not a harddrive bottleneck issue.

-Maybe this is apache related?
-Or network related?

Here is sar -n ALL output for the evening hours. According to this we are not sending more than 400kb/s over 10 minute averages. Our bandwidth is 10mbps, so we are not even using half of it yet.

                IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
08:10:01 PM        lo      0.63      0.63     66.48     66.48      0.00      0.00      0.00
08:10:01 PM      eth0    342.69    431.72  59755.43 368330.14      0.00      0.00      0.00
08:10:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:10:01 PM      sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:20:01 PM        lo      0.43      0.43     45.08     45.08      0.00      0.00      0.00
08:20:01 PM      eth0    310.96    389.32  62960.35 327811.74      0.00      0.00      0.00
08:20:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:20:01 PM      sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:30:01 PM        lo      0.68      0.68     71.97     71.97      0.00      0.00      0.00
08:30:01 PM      eth0    341.29    421.12  69692.70 354844.36      0.00      0.00      0.00
08:30:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:30:01 PM      sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:40:01 PM        lo      0.79      0.79     87.60     87.60      0.00      0.00      0.00
08:40:01 PM      eth0    365.76    451.49  66813.87 385379.38      0.00      0.00      0.00
08:40:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:40:01 PM      sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

08:40:01 PM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
08:50:01 PM        lo      0.47      0.47     49.49     49.49      0.00      0.00      0.00
08:50:01 PM      eth0    336.77    413.37  57303.78 357556.44      0.00      0.00      0.00
08:50:01 PM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:50:01 PM      sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00


08:00:01 PM    totsck    tcpsck    udpsck    rawsck   ip-frag
08:00:01 PM       218        21         9         0         0
08:10:01 PM       200        21         9         0         0
08:20:01 PM       211        22         9         0         0
08:30:01 PM       205        22         9         0         0
08:40:01 PM       191        21         9         0         0
08:50:01 PM       201        21         9         0         0
share|improve this question
add comment

3 Answers 3

  • You can check whether another disk would help, if you serve up some of the files from a ramdisk / memory fs. Nothing beats memory speed.
  • Calculate your download speed times your concurrent downloads. Perhaps some your upstream connection has not enough bandwidth.
  • Check from a third (fourth, fifth) location to make sure you're not hitting a limitation of the download location.
share|improve this answer
    
Just added vmstat output thinking it might help. I'm thinking this image should stay in memory since it is called 6 times an hour. I'll test the upstream connection per advise. Before trying out a new harddrive, is there a way to check if the harddrive is the bottleneck? –  Haluk Nov 26 '10 at 22:09
    
Create a fs in memory. Use it as a symlinked dir inside your htdocs. Then you don't have to rely on the assumption that the file from the disk is cache. –  knitti Nov 26 '10 at 22:12
    
That's good advise. Thank you. I've setup the memory directory. Evening is already past here. We should get the results tomorrow evening. –  Haluk Nov 26 '10 at 22:59
add comment

Sounds like a network issue. Do a packet capture of the image download during both non-peak and peak hours. Compare the two and see if you are getting a lot of RST's.

share|improve this answer
    
I just ran a stress test and did a packet capture. I can see some RSTs but they are far and few. –  Haluk Nov 28 '10 at 15:49
    
Interesting. Have you tried replicating the issue from a different remote site? –  sdanelson Nov 29 '10 at 2:47
    
Yes, this used to happen from home as well. Sometimes image delivery from the site would slow down. So I decided to have it routinely monitored from an alternative site which is, hopefully, an expert on monitoring. –  Haluk Nov 29 '10 at 16:24
add comment

You need to prioritize the TCP ACK packets. See http://phix.me/dm/

share|improve this answer
    
I read the link you shared. Thank you. Having read it, it looks like changing packet priorities should be a final resort. Before that there should be several things I can do at the OS level. –  Haluk Nov 29 '10 at 16:23
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.