Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a GWT application running on Tomcat on a cloud linux(Ubuntu) server, recently I released a new version of the application and suddenly my server response times have gone from 500ms average to 15s average. I have run every monitoring tool I know.

  • iostat says my disks are 0.03% utilised
  • mysqltuner.pl says I am OK other see below
  • top says my processor is 99% idle and load average: 0.20, 0.31, 0.33
  • memory usage is 50% (-/+ buffers/cache: 3997 3974)

mysqltuner output

[OK] Logged in using credentials from debian maintenance account.

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.1.63-0ubuntu0.10.04.1-log
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster 
[--] Data in MyISAM tables: 370M (Tables: 52)
[--] Data in InnoDB tables: 697M (Tables: 1749)
[!!] Total fragmented tables: 1754

-------- Security Recommendations  -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 19h 25m 41s (1M q [28.122 qps], 1K conn, TX: 2B, RX: 1B)
[--] Reads / Writes: 98% / 2%
[--] Total buffers: 1.0G global + 2.7M per thread (500 max threads)
[OK] Maximum possible memory usage: 2.4G (30% of installed RAM)
[OK] Slow queries: 0% (1/1M)
[OK] Highest usage of available connections: 34% (173/500)
[OK] Key buffer size / total MyISAM indexes: 16.0M/279.0K
[OK] Key buffer hit rate: 99.9% (50K cached / 40 reads)
[OK] Query cache efficiency: 61.4% (844K cached / 1M selects)
[!!] Query cache prunes per day: 553779
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 34K sorts)
[OK] Temporary tables created on disk: 4% (4K on disk / 102K total)
[OK] Thread cache hit rate: 84% (185 created / 1K connections)
[!!] Table cache hit rate: 0% (256 open / 27K opened)
[OK] Open file limit used: 0% (20/2K)
[OK] Table locks acquired immediately: 100% (692K immediate / 692K locks)
[OK] InnoDB data size / buffer pool: 697.2M/1.0G

-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    MySQL started within last 24 hours - recommendations may be inaccurate
    Enable the slow query log to troubleshoot bad queries
    Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
    query_cache_size (> 16M)
    table_cache (> 256)
share|improve this question

1 Answer 1

If anyone is interested, the problem was that we had at some point exceeded the maximum number of connections in the connection pool which just caused a snowball effect at threads waited for a connection from the pool. This explains why the machine was 'asleep', everything was waiting in a queue for a connection to become available.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.