Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am currently part of a team working on a new large-scale project consisting of a web platform with content consumed through an API by various mobile devices. We really anticipate a lot of traffic during the first months of launch so really good planning on capacity, scalability, redundancy and availability of the services is a must.

I'd like some reading recommendations (web, books) on matters regarding good practices about server architectures like where to put the web server, where to put the database, where to put the mail server, how to make master/slave configurations, caching, load balancing and stuff like that.

I don't really want to dig deep into the subject since I am just the guy who came up with the concept but also the leading developers on one of the mobile platforms. I have pretty solid experience with basic systems administration but I'll have to expand my knowledge on the topic so I can guide the platform development team and plan hosting requirements.

Also, is there some kind of independent consultants specializing on stuff like that? In case I find myself in trouble.. :P

share|improve this question
add comment

4 Answers

up vote 2 down vote accepted

For capacity planing you probably want to read the nice works of Neil J. Gunther. Especially his new work is really excellent:

or John Allspaw's

and for the operational POV:

Allspaw is the guy who's in charge of operations at flickr.

General books about high availability, disaster recovery and distributed systems should also be easy to find on amazon or where you get your books from. It really helps a lot to understand the general problems so that you guys can understand the particular solutions.

Never ever underestimate proper application architecture and please do yourself and your team a favor by implementing instrumentation in your application - can't stress this one enough.

Be aware that for any of the things you asked literature for are topics that need to be handled from people that are experts on exactly that topic as all of them are extremely broad and need deep expertise to be implemented properly. Also they are nothing like sysadmin fundamentals or basics. There's also a big dependence on the exact product that you're going to use!

Regarding the consulting companies - name your products or describe your architecture (if possible) and I can recommend some companies that I worked with on a similar project as an infrastructure consultant.

share|improve this answer
    
Thanx for your kind and complete response buddy, I really appreciate it. Following the "premature optimization is the root of all evil" quote, I only want to architect the launch and critical mass/traffic infrastructure, that's why I only need some basic understanding of the modern ways to things. If the project goes well, I guess we are gonna hire specialists for the purpose of scaling. So basically we are talking about a mobile location based dating service. I am thinking of a beast dedicated server with S3 storage for static content (images) –  epron Dec 16 '10 at 12:29
    
We are also thinking of mongoDB for our database solution since the geospatial and sharding capabilities might come handy. Another option is use the dedicated server only for web and API serving and use Amazon RDS for db and S3 for storage and db backups. I am concerned of network latency between the dedicated and RDS though... –  epron Dec 16 '10 at 12:41
    
The DB architecture is one of the main corner stones of the app and you need to get that right from the start on, but I see that you are basically asking the right questions since you're thinking of things like sharding or even going the NoSQL route. Amazon is generally a very nice solution since you can scale that beast quite easily (also in terms of €€€) but your application then needs to handle all the availability aspects - depending on the ppl u have at hand for the job that actually might be good thing. Scaling can't be bolted on afterwards but I guess u've figured already. –  pfo Dec 16 '10 at 13:05
    
Also keep in mind that you're probably not building the next facebook and don't have to handle 500e6 users. –  pfo Dec 16 '10 at 13:09
    
So, dear friend, you think that having the beast (bit.ly/e1b2OC) serving Web and API calls, an S3 instance handling static content and Amazon RDS as a database is a good starting architecture? I was thinking maybe starting of with the database running in the beast (considering the high performance configuration of this server) but I am having concerns about network latency for queries travelling back and forth from Amazon. –  epron Dec 16 '10 at 14:20
show 2 more comments

I would highly recommend reading Oreilly's Beautiful Architectures (More focused on the actual programs that will be running) and Cloud Application Architectures (more focused on the system itself) .

Here are the links:

http://oreilly.com/catalog/9780596517984/

http://oreilly.com/catalog/9780596156374/

Hope that helps, RayQuang

share|improve this answer
add comment

Yes, there are consultants who specialize in infrastructure design and implementation.

What kind of page views are you looking to hit each month? Having a successful launch is of course the most critical aspect of a project and i'd be interested in learning more about what you guys are working on.

In order to handle this, a lot of factors go into play. database design, load balancers, hardware specifications, network performance, cdn network, etc...

Do you already have an infrastructure in place? The design reference would most likely be leverage by professionals with ITIL. If there are any compliance regulations such as credit cards being transmitted over a mobile platform PCI compliance would be a factor, or HIPAA for the healthcare records.

Shoot me an email and i'd be happy to go this with you.

share|improve this answer
add comment

I have mentioned this book several times in other posts because it is one of the most completely thorough and information-dense books on the topic: Theo Schlossnagle's Scalable Internet Architectures covers a lot of the scalability arena in depth from a generalized systems perspective. Lots of really good general sysadmin stuff in here as well.

Would like to second the mention of John Allspaw's Art Of Capacity Planning and especially Cloud Application Architectures as CAA deals with Amazon EC2 directly (provides solutions and examples using EC2 infrastructure and API tools).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.