We have a MySQL-based application that is very response time sensitive. A MySQL stored procedure needs to respond on average in less than 10 milliseconds and the maximum acceptable response time is 50 milliseconds. We are hitting the average response time with flying colors (more like 3 milliseconds) but over a five hour load test we will get some (~50) responses greater than 50 milliseconds with a maximum response time sometimes as high as 265 milliseconds.
There is a lot of information out there about tuning MySQL and Linux for optimal performance, and I believe we've applied most of the best practices, but I can find very little about how to reduce the incidence of outliers and reduce maximum response time.
How can I diagnose the cause of these outlier response times and what Linux and MySQL tunings can I apply to reduce them?