Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I store a huge amount of reporting elements in a MySQL database. These elements are stored in a simple way :

KindOfEvent;FromCountry;FromGroupOfUser;FromUser;CreationDate All these reporting elements should permit to display graphs from different points of view. I have tried using SQL requests for that but it is very slow for users. As this graph will be used by non-technical users, I need a tool to pre-work the result. I am very new to all this data-mining, reporting, olap concepts. If you know a pragmatic approach not so time consuming, or a tool for that, it would help !

share|improve this question

1 Answer 1

The pragmatic, non-time-consuming approach would be to hire a data mining expert. Even getting one temporarily should be enough to get a good system in place and get you up to speed.

SQL needn't be slow. If you run your SQL query with EXPLAIN EXTENDED in front of them, MySQL will tell you useful information about what indexes it can use, what indexes it will use and what percentage of rows were filtered out by each index. MySQL query and index tuning is a large topic. If you have specific queries and indexes you want help tuning, the guys over at DBA.SE just love doing that stuff.

You can also pre-process the data and store the results in a separate table which is used by the front-end reporting tool. Something like a cron job or a queue scheduler can be used to process new data regularly as long as the users are aware that the data may be out of date. A little note next to the chart explaining that "This data was calculated at 10:39AM, Thursday 29th March, 2012" should be adequate.

If you have truly exceeded the limits of MySQL, you could try spreading out the load over multiple machines using MapReduce. Hadoop and MongoDB both provide MapReduce. Tutorials are available.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.