I'm working in the early stages of development of a Foursquare mashup. We get some information using the 4SQ API, but store our own "checkins" in our infrastructure (kind of the 4SQ "Checkin" but different in some points).
I'm in the backend part. PHP and MySQL, so far. Just starting. Works fine. The clients are iPhone/Android. In a few weeks, we'll perform some testing with growing number of users. Of course, we'll have to scale the infrastructure out. We count with Amazon RDS access.
Those days I'm reading documentation about datastores, scaling, sharding, etc. Things I've learnt (correct me if I'm wrong):
- NoSQL is not a solution for all possible scenarios (compromise between Availability, Consistency and Partition Tolerance).
- SQL & NoSQL better than SQL | NoSQL (i.e.: Hadoop + MySQL)
- Memcached is a good tool to cache database results.
- Redis and FlockDB are good tools for social graphs (friendship is a concept we use apart from 4SQ too).
- Cassandra is a good tool to store feeds/timelines.
- Scribe is a good tool to store log data.
- PIG is a good tool to get data reports.
- MongoDB has several interesting features (the use of JSON, geospatial indexing, being considered the "MySQL of the Non-relational world", fast insertions, etc). But I'm not sure about the kind of application MongoDB is really good at.
I find all this stuff really exciting, but there's too much information. I'm just a developer, but I will probably be part of the team to make important decisions about how to scale our application out. So, I want to put my DevOp Hat on, but I'm a bit lost.
Should I take any previous decision at code level (PHP, MySQL configuration)? What could be the most critical parts of a 4SQ-like system, with hundreds (potentially thousands) of checkins per minute?
Many thanks in advance