I'm late to the party here; but I have something to contribute...
Cal Henderson talks (briefly) about the issues you're facing in his book Building Scalable Websites. You should read through the chapter on web service APIs.
As easel correctly points out, reverse proxy caching will typically not give much benefit for an API.
Two things that will benefit you are:
- A distributed key-value cache for the data that you're working with. I.e. you can cache your hot data in your platforms native object format, or as simple serialized arrays, or whatever else is fast for your platform. This limits the number of hits against you database, where you'll have your worst scaling problems. Your API servers will still need to fetch any non-cached datasets, and to serialize the response -- but at least the serialization part is only CPU bound, and can be scaled easily.
- Some sort of rate-limiting system, i.e. the capacity to throttle or deny clients who are making too many requests. API calls will typically involve fairly heavy processing on your servers (as they read or manipulate raw data), so protecting yourself against badly written clients makes sense.
In addition to the above, Cal Henderson suggests creating open source client libraries in popular languages, and putting best practices into these (i.e. client-side caching, rate limiting). This way 3rd party developers will have an easy, compliant code platform to re-use and build upon. The idea sounds great IMHO, but also somewhat costly.
