Everyone else's opinions all have their own merits. I would argue that if all (or majority) of your apps are LAMP stack-ish, I'd say cluster the database portion (MySQL/PostGreSQL/etc.) either active-active/failover however you want.
In terms of application performance, you really need more metrics to prove a case for any kind of clustering of sorts. To add the complexity without regard to actual measurements let alone the possible manageability nightmare, make sure you have some proof.
From a logical standpoint, if you're just launching new sites/apps and databases are always a part of each app, I'd argue that clustering the databases together would be beneficial from a management standpoint and ultimately would allow you to virtualize the webservers (lowering future costs). Keeping both layers independent will help you in the following ways:
Data above all else the most important thing in IT. Lose info; lose customers which leads to losing money. A cluster could improve performance/availability/redundancy/manageability if done correctly (and tested vigorously).
Virtualizing your webservers allows for availability and manageability. If a virtual server goes down, the webserver could migrate to another viable virtual host and the database cluster could keep humming along. Plus, less to purchase for each new web app that needs to be installed. Wouldn't it be a lot cheaper to add more RAM to a Virtual Host rather than buying a whole new server?
So to answer your questions directly..
This situation lead me to think:
shouldn't we create a cluster, instead
of adding individual servers for each
app?
Yes, to some extent. Unless you're hosting one big webapp and performance is a major issue (like.. facebook?), clustering the webservers and maybe including some caching servers (memcached) for performance would help out definitely. But in your case, it sounds like you have lots of little or simpler sites/webapps with databases for each app. If load/performance isn't a problem here, virtualization will help with the uptime/availability.
From an uptime/redundant/management standpoint, I would definitely cluster the database servers in a solid group of servers. Keep the data safe, and lower your odds of downtime/disaste. That's much more important than a website as a site is simply just a bunch of files being served with apache, nginx (pick your webserver). Having a backup of a site isn't much in terms of storage (usually) and restoring a site should be fairly simple and straightfoward. Or you could get really creative and when all your sites are running in VMs, backup the VMs for any change on the webserver, back them up and if the running/production image becomes broken or difficult to fix, simply copy the good copy from archive. The database cluster should still be running and once the new VM is up and running, it should be pointing to the db cluster. Just an idea. I bet other people have better ideas/techniques, but whatever. It's just a thought.
I can attest that, while some servers
need better hardware (RAM, mostly),
some other ones have been idle for no
less than three weeks, so with a
cluster we could use those resources
better.
If you're jumping to the conclusion that you need better hardware, you better figure that out first before building a cluster around bad parts. If you're having numerous, similar problems crop up on a consistent basis, a cluster won't really fix anything. Yes a cluster will adjust to the issues it was designed to but get to the root problem first. Why build on a bad foundation? But if your point is not having enough hardware, that's a different problem which can be out of your hands. I'd look into schmoozing whoever does the budget heavily.
At the same time, I'm not sure the
results are worth the time and effort
needed to put together and mantain the
aforementioned cluster (I always
wanted to use the word
"aforementioned").
Aside from your bubbling vocabulary, it's good to see that you're at least realizing the weight of the quest that lays before you should you choose to go the cluster route. I'm going to take a wild stab and say I think you're looking for the reason "to" go ahead with the cluster and while many people here may agree with you, only you know your technical strengths and weaknesses. 10 webservers isn't too bad but before too long when you get more and more webservers to host, between managing the servers, backing the up, dealing with uptime/availability and redundancy you're going to have to do something. If you don't do something now, that might be fine, but I would start investigating all your options for a date in the future. Who knows? Six months from now you'll add 10 more apps and your perspective changes drastically.
From my experience before doing any such adventure, do research, investigate your current assets, build a strategy/plan and fill in all the gaps in that plan. In fact here's a great document should you start researching clustering and such: http://www.scribd.com/doc/4069180/Caching-Performance-Lessons-from-Facebook. Granted that something always goes wrong in a plan, the more time you have to review your plan and revise you'll catch possible pitfalls.
