Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We have some fairly large datasets (user events and server log information - >100 GB) that's becoming fairly unwieldy for data processing. I've seen lots of activity around NoSQL/Hadoop/etc and I was wondering what SV had to say about a paired solution. The absolute ideal situation would be:

  • A "master-slave" like synchronization between our live MySQL and the NoSQL/Hadoop servers, but different enough that we can build custom indexes/etc.
  • Ability to run standard aggregation results on the NoSQL/Hadoop in trivial time such as:
    • <1 sec for sum(*) where event_type = 'blarg' between 'date 1' and 'date 2'
    • Give me all the incoming search terms (which we record) for this page and the children of this page over an arbitrary time period and their count
  • A simple (<10 minute) way to update a developer's machine.

Thoughts? We've tried a number of solutions around MySQL and nothing meets all these elegantly.

share|improve this question
Why do you want to keep the MySQL side? – Erik Aug 23 '10 at 15:10
The entire app is based on that - we just want to use this for mining. – aronchick Oct 6 '10 at 21:51
up vote 2 down vote accepted

MongoDB is simple and now has auto sharding. It is not very efficient for disk usage, so you will need to give it a lot of disk space. It can handle queries, but it will need indexes unless you want it to scan billions of records. What we have done is to actually store summaries in Mongo because if you know your query you can build an optimized data structure around it and be extremely efficient.

Have you tried Infobright with MySQL? It does automatic compression and is FAST. It could be enough for you.

share|improve this answer

You need to code some kind of adapter by yourself!

share|improve this answer
is there really no better way? – aronchick Jul 31 '10 at 4:51

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.