We have some fairly large datasets (user events and server log information - >100 GB) that's becoming fairly unwieldy for data processing. I've seen lots of activity around NoSQL/Hadoop/etc and I was wondering what SV had to say about a paired solution. The absolute ideal situation would be:
- A "master-slave" like synchronization between our live MySQL and the NoSQL/Hadoop servers, but different enough that we can build custom indexes/etc.
- Ability to run standard aggregation results on the NoSQL/Hadoop in trivial time such as:
- <1 sec for sum(*) where event_type = 'blarg' between 'date 1' and 'date 2'
- Give me all the incoming search terms (which we record) for this page and the children of this page over an arbitrary time period and their count
- A simple (<10 minute) way to update a developer's machine.
Thoughts? We've tried a number of solutions around MySQL and nothing meets all these elegantly.