I'm currently evaluating whether to use RightScale to manage a production environment in EC2. I intend to use Puppet for configuration management either way (the declarative approach seems far better than running scripts), am running a somewhat nonstandard stack (e.g. MongoDB), and am uncertain about how much value RightScale would add relative to Puppet + Amazon's auto-scaling + another hosted monitoring system. Those who use RightScale, what features do you find important? Is its auto-scaling support (including keeping single instances running) more powerful than Amazon's?
I think I can help with some of the differentiation. Full disclosure, I work for RightScale.
You are making a good decision to abstract your configuration management from base images--it will preserve your future infrastructure choice. However, when you start adding cloud specific tools like CloudWatch+Autoscaling, you are beginning to lock yourself in. RightScale's dashboard, API and multi-cloud images are AWS-region and cloud agnostic, allowing you to preserve future choice and flexibility, and everything about your system definitions, the monitoring tools, and the RightLink server agent you put on your machines to enable them are open source. On average we see a 60% time savings getting started, 50% on ongoing maintenance of systems, and over 90% when reusing your existing assets for future projects as well admin:server ratios of up to 1:10,000+.
Puppet+RightScale works fine and you may be interested in this support article: http://support.rightscale.com/index.php?title=06-FAQs/FAQ_0181_-_Does_RightScale_support_Puppet%3F There are still good reasons to create script based ServerTemplates for the PuppetMaster and PuppetClient (makes them easy to maintain later), however you can use manifestos for the configuration of the rest of your environment. You can also use standard scripts in compliment and it's sometimes easier to do that for operational actions. Clustering these server configurations into deployments will help you to better define, operate, clone, and track resources allocated to specific systems.
RightScale's autoscaling is fundamentally different from AWS--which can be better defined as autolaunching since it will only trigger the AMI itself to spawn. Server Arrays (server definitions that scale horizontally), can be triggered in 3 ways. Monitoring, which includes over 60 metrics out of the box plus the ability to create your own (collectD), can trigger alert actions for system remediation including self healing (reboot/relaunch), script execution, or scaling. When scaling based on monitoring data we use a vote based trigger to prevent a single node in a cluster from provisioning or deprovisioning resources. This also means that nodes are not being watched as a cluster either so you can be notified when a single server is having a problem (CloudWatch looks at the aggregate). In compliment to load you can also schedule scaling if you have predictable patterns. Alternatively you can define an array to scale based on a job queue (good for embarrassingly parallel batch processing tasks).
Aside for automation and asset management, there are many other reasons to use RightScale that generally are not apparent when considering using cloud infrastructure initially, but can end up being extremely important to an organization. Role based security permissioning for your account users, infrastructure security audit reports, managed SSH login rules, cost allocation/quotas and tracking, and multi-credential/enterprise group management are a few.
Your software stack is not a problem. We have a lot of experience with MongoDB in other customers like GameTree.TV, although there are no prebuilt ServerTemplates in the library for it at this time like there are for other NoSQL solutions like Couchbase.
Please try out some of these tools in the Developer Edition if you have not already, you can sign up at my.rightscale.com. You can also reach me at matt [at] rightscale or post further questions here for the group. There are also some interesting posts on Quora (http://www.quora.com/RightScale/What-have-startups-experiences-been-with-RightScale). Best of success with your project.
I work at Scalr, an open-source cloud management software competing with RightScale. Scalr does not support Puppet at this time but Chef, one of its competitor, is integrated in our software. So I guess your question is still relevant in our case and I would like to share my experience.
Cloud management softwares provides a lot of automation tools that are not included in configuration software Chef and Puppet for example. When we started out building Scalr, we were dead set on auto-scaling. But we quickly discovered that there were much more painful problems to solve. This list was written by Sebastian Stadil and explains well the additional value of using a Cloud Management Software:
You are spending a ton of time on repetitive tasks.
Cloud management software takes the repetition away from managing infrastructure– like mounting that 4th ebs volume on every one of your 12 database servers. You operate at a higher level by defining what components your server farm is composed of, and what each server in it looks like. Cloud management software then applies that to the actual infrastructure. This way you don't have to repeat yourself: you can manage 10,000 servers as easily as 10. It brings automation.
You don't know where things are, or what they correspond to.
Cloud management software keeps you organized. Cloud infrastructure is constantly changing. Servers are constantly added, removed, or replaced. Same goes for storage volumes, ip addresses, snapshots, etc. This makes it nigh impossible to keep your system of record up to date. Cloud management keeps things tidy.
You don't know how much your applications are costing you.
Cloud management software gives you visibility and helps you keep a budget. How much are you spending on that data processing farm? How much savings did that new caching tier bring you? Has your R&D group gone overbudget? Everyone wants to limit waste and abuse, but it's hard to do so unless you have the right tools.
You want to make sure your colleagues and contractors don't leave the company with the keys to your infrastructure.
Cloud management software helps you manage permissions granted to your co-workers. It's a bad practice to give everyone root access to all your servers, to everyone in your organization. Especially if you have contractors and interns.
You want to make sure your infrastructure is secure.
Cloud management software gives your security team auditing tools to spot vulnerabilities. Whether it's as simple as logging who opened port 22 on your database, or enforcing security policy across large and diverse infrastructure.
You want to experiment with some architectural changes.
Cloud management software makes it easy to try new architectures. Want to try out a new front-end reverse proxy? Add a caching tier? Move a mysql table to ramdisk? Introduce a key-value database? It's all easy and doesn't require spending days configuring things.
You want a Plan B in case disaster strikes.
Cloud management software helps plan for failure, and getting that Plan B ready. Whether is simply scheduling backups at regular intervals and rotating them, or a complete cross-cloud disaster recovery plan, it gives you tools and acts as preventative medecine.
You want to visualize your infrastructure.
Cloud management software provides data visualization and presentation. What performance are you getting from your servers? Are there any outliers? When integrated into your monitoring tier, you can get the visibility that managers dream of.
You want to version your infrastructure.
Cloud management software allows you to version your infrastructure the same way you do with your code. It allows you to express your infrastructure as code, so you can create exact copies for testing, revert to a previously functioning system, and anything in between.