Server HTTP Load times slow?

Question

My server @ codemeh.com (HTTP Server) seems to be randomly loading slowly, I cannot tell if it just my forums (http://www.codemeh.com/forums/) that are loading slowly or if the WHOLE site is just loading slowly since my forums are the largest thing on the site right now.

load average: 0.02, 0.17, 0.20

That is super low to my knowledge. I have tried Google Page Analytic plug-in for FireFox to solve the problem but nothing comes up that is VERY bad. If someone could investigate this for me since I am very new at apache and server configurations. Thanks!

(top):

    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                      
 7493 www-data  15   0 98.2m  16m 9092 S    3  0.8   0:27.24 apache2                                      
26429 www-data  15   0 98.2m  15m 7392 S    3  0.7   0:03.45 apache2                                      
26477 www-data  17   0 98.2m  15m 7396 S    3  0.7   0:03.16 apache2                                      
    1 root      15   0  2468 1384 1156 S    0  0.1   0:00.49 init                                          1367 root      25   0  2564  816  660 S    0  0.0   0:00.00 xinetd                                       
 1526 root      15   0 29576 5420 1976 S    0  0.3   1:02.69 fail2ban-server                               3703 root      15   0 13512 9312 1696 S    0  0.4   0:11.59 miniserv.pl                                  
 3915 postfix   15   0  6056 1652 1320 S    0  0.1   0:00.00 pickup                                        4010 root      15   0  4548 1296  972 S    0  0.1   0:37.36 ntpd                                         
 7448 root      15   0 98528  26m  20m S    0  1.3   0:00.27 apache2                                       7454 www-data  18   0 33580 2616  368 S    0  0.1   0:00.04 apache2                                      
 7528 www-data  18   0  108m  24m  15m S    0  1.2   0:27.60 apache2                                       7974 root      16   0  8700 2728 2164 S    0  0.1   0:00.08 sshd                                         
 8123 cdog5000  15   0  8832 1596  896 S    0  0.1   0:00.00 sshd                                          8126 cdog5000  18   0  4484 1716 1384 S    0  0.1   0:00.00 bash                                         
 8141 cdog5000  15   0  2344  980  796 R    0  0.0   0:00.11 top                                          13461 root      15   0  8700 2728 2164 S    0  0.1   0:00.07 sshd                                         
13567 cdog5000  18   0  8832 1492  896 S    0  0.1   0:00.33 sshd                                         13569 cdog5000  18   0  4484 1728 1388 S    0  0.1   0:00.09 bash                                         
17983 root      15   0  4392 1268  988 S    0  0.1   0:00.00 su                                           17987 root      15   0  4516 1752 1380 S    0  0.1   0:00.09 bash                                         
18081 www-data  15   0 98.2m  14m 6588 S    0  0.7   0:04.91 apache2                                      20000 www-data  15   0 98.3m  15m 8040 S    0  0.8   0:02.45 apache2                                      
20019 www-data  15   0 98.2m  14m 6808 S    0  0.7   0:04.97 apache2                                      30343 root      15   0  3964 1012  764 S    0  0.0   0:00.03 vsftpd                                       
30382 root      15   0  2304  908  716 S    0  0.0   0:00.62 cron                                         30401 mysql     17   0  141m  17m 5416 S    0  0.9   1:02.20 mysqld                                       
30424 root      15   0  5472  912  504 S    0  0.0   0:00.04 sshd                                         30473 syslog    15   0  1916  676  536 S    0  0.0   0:01.02 syslogd                                      
30611 amavis    15   0 33872  25m 2292 S    0  1.2   0:03.11 amavisd-new                                  31890 amavis    18   0 34888  24m 1792 S    0  1.2   0:00.00 amavisd-new                                  
31891 amavis    18   0 34888  24m 1784 S    0  1.2   0:00.00 amavisd-new                                  32397 clamav    18   0  104m  84m 1272 S    0  4.1   1:06.46 clamd                                        
32563 clamav    15   0 12832 5716 4440 S    0  0.3   0:01.29 freshclam                                    32573 root      23   0  1892  456  372 S    0  0.0   0:00.00 courierlogger                                
32575 root      18   0  2096  684  544 S    0  0.0   0:00.01 authdaemond                                  32583 root      23   0  1892  360  284 S    0  0.0   0:00.00 courierlogger                                
32584 root      24   0  2000  612  516 S    0  0.0   0:00.00 couriertcpd                                  32598 root      23   0  1892  360  284 S    0  0.0   0:00.00 courierlogger                                
32599 root      25   0  2000  612  516 S    0  0.0   0:00.00 couriertcpd                                  32604 root      18   0  1892  460  372 S    0  0.0   0:00.00 courierlogger                                
32605 root      18   0  2000  624  532 S    0  0.0   0:00.00 couriertcpd                                  32607 root      18   0  2308  404  256 S    0  0.0   0:00.02 authdaemond                                  
32608 root      18   0  2096  260  116 S    0  0.0   0:00.03 authdaemond                                  32609 root      15   0  2308  404  256 S    0  0.0   0:00.03 authdaemond                                  
32610 root      18   0  2096  260  116 S    0  0.0   0:00.02 authdaemond                                  32612 root      18   0  2308  404  256 S    0  0.0   0:00.02 authdaemond                                  
32621 root      24   0  1892  364  284 S    0  0.0   0:00.00 courierlogger                                32622 root      25   0  2000  608  516 S    0  0.0   0:00.00 couriertcpd                                  
32633 root      15   0  105m  936  716 S    0  0.0   0:02.26 nscd                                         32719 root      16   0  6252 1680 1344 S    0  0.1   0:01.24 master                                       
32738 postfix   15   0  6188 1776 1400 S    0  0.1   0:00.44 qmgr                                         32758 postfix   15   0  6492 2564 1788 S    0  0.1   0:00.14 tlsmgr      

(/etc/apache2/sites-available/default):

NameVirtualHost *
<VirtualHost *>
        ServerAdmin webmaster@localhost

        DocumentRoot /var/www/web1/web/
        <Directory /var/www/web1/web/>
                Options Indexes MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

</VirtualHost>

I have fail2ban server and I dont have any firewall at this point and time that I know of. SMF is 2.0 RC4 and apache version is 2.2.14. I run a MySQL server on another box in the same DC (Persistent Connection). I installed eAccelerator today and it didnt help.

Answer 1

A couple things.

1) Yes once page is generated and sent to the browser, other links will download in parallel to an extent. IIRC, most browsers will only make 2 requests to the same domain at a time. This means if you have 40 images, css, js, etc from the same domain, they will download 2 at a time, until it's all downloaded. Perhaps look into moving static content to a subdomain.

2) I noticed that at least one of your images is 126KB (the body_bg). That's a gigantic number for a website. Try to optimize the file size better

3) I noticed this in the request: "Keep-Alive timeout=15, max=98". If you have access Try reducing the keep-alive. From http://www.howtoforge.com/configuring_apache_for_maximum_performance , "Set this to a low value, perhaps between two to five seconds. If it is set too high, child processed are tied up waiting for the client when they could be used for serving new clients."

Hope this helps

Answer 2

Your site doesn't seem too bad in terms of loading time (typically 3-4 seconds with a few 6-7 seconds due to external files) but here are a few suggestions:

• Make sure you are running a PHP opcode cache (ex: APC, eAccelerator). This can significantly decrease the script execution time.
• Look into using mod_expires for files that don't change often (ex: JavaScript, CSS , images, etc...). This can reduce the number of requests a browser has to make.
• Also look into mod_deflate for compressing text based files (PHP, JavaScript, CSS). This can reduce the file sizes and thus reduce the page loading time. Note that they will use up some CPU so evaluate their performance and resulting server load.
• A good half, or more, of your page loading times, files, and total bytes comes from external sites (Google, Affinity). If don't need, or can optimize these, your pages would load considerably faster.
• The FireBug / YSlow extensions for FireFox are invaluable for debugging these sort of things. Run the YSlow extension for many other suggestions to increase page speed (note that not all suggestions YSlow makes may be applicable to you).

I think just doing all/some of these can get your page load average below 1 second without having to look at any caching layer like Squid/Varnish (which could reduce it further with more work if desired).