Apache performance help

Question

I've had performance issues recently with my (Linux centOS based) web server. I've had pretty slow response times for all sites on the server, whereas previously, page loading speeds were much better. I have eAccelerator running, gzipped content, optimised mysql queries and other standard optimisation processes which I've made sure have been implemented. I've had a look at the apache stats to see if there is something going on there in terms of performance, is there anything there I should be concerned about (e.g request rate)? Grateful for any help as I'm not experienced in this area!

Maximum connect burst length: 0

Total: connections 1 requests 1 replies 1 test-duration 1.512 s

Connection rate: 0.7 conn/s (1511.9 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 1511.9 avg 1511.9 max 1511.9 median 1511.5 stddev 0.0
Connection time [ms]: connect 32.3
Connection length [replies/conn]: 1.000

Request rate: 0.7 req/s (1511.9 ms/req)
Request size [B]: 79.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 1409.5 transfer 70.2
Reply size [B]: header 262.0 content 17425.0 footer 2.0 (total 17689.0)
Reply status: 1xx=0 2xx=1 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.35 system 1.02 (user 23.1% system 67.5% total 90.6%)
Net I/O: 11.5 KB/s (0.1*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

Edit: I've also run mysqltuner on the server to see if there is anything there which can be improved - couple of things I can do.

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.92-community
[!!] Switch to 64-bit OS - MySQL cannot currently use all of your RAM

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB +Federated +InnoDB -ISAM -NDBCluster 
[--] Data in MyISAM tables: 111M (Tables: 662)
[--] Data in InnoDB tables: 10M (Tables: 468)
[!!] Total fragmented tables: 114

-------- Security Recommendations  -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 4d 7h 19m 52s (2M q [5.717 qps], 94K conn, TX: 3B, RX: 208M)
[--] Reads / Writes: 82% / 18%
[--] Total buffers: 58.0M global + 4.4M per thread (400 max threads)
[OK] Maximum possible memory usage: 1.8G (84% of installed RAM)
[OK] Slow queries: 0% (22/2M)
[OK] Highest usage of available connections: 3% (14/400)
[OK] Key buffer size / total MyISAM indexes: 16.0M/63.2M
[OK] Key buffer hit rate: 99.5% (1M cached / 8K reads)
[OK] Query cache efficiency: 91.9% (1M cached / 1M selects)
[!!] Query cache prunes per day: 17638
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 43K sorts)
[!!] Temporary tables created on disk: 48% (205K on disk / 420K total)
[OK] Thread cache hit rate: 99% (14 created / 94K connections)
[!!] Table cache hit rate: 3% (1K open / 28K opened)
[OK] Open file limit used: 51% (1K/2K)
[OK] Table locks acquired immediately: 99% (184K immediate / 184K locks)
[!!] InnoDB data size / buffer pool: 10.4M/8.0M

-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    Enable the slow query log to troubleshoot bad queries
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries without LIMIT clauses
    Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
    query_cache_size (> 16M)
    tmp_table_size (> 16M)
    max_heap_table_size (> 16M)
    table_cache (> 1024)
    innodb_buffer_pool_size (>= 10M)

Answer 1

Is your system depending on multiple database transactions?
If you really have all performance optimizations in place and your site becomes slower, it might also be database which is growing larger and user base which is frowing proportionally with the database (e.g your sites become more and more popular), and needs more time to run the same processes, you execute on the smaller database.

For an average site, when database grows 200MB+ its performace slowly decreases asking for more memory and a bit different Indexing strategy.

You can use tools like Pingdom (they have free accounts) to monitor uptime and responce times for your site, and then you an find bottlenecks, evaluate whats happening. Also, what is the average load on your system?

How many visitors do you have per day, and how many did you had one year ago? There might be difference.