Cloud or VPS for a data-mining proof-of-concept setup?

Question

We need to host a proof-of-concept system online for an I/O heavy data-mining app. We typicall need 100-200GB per system, but we have low CPU and bandwidth requirements. Ideally, we'd like to have a customized Linux VM that we can clone, load the data, run it online for a few weeks/months (as long as the customer wants to play with it) and then take it down. We're OK with getting charged for hosting the VM template, but ideally we'd like to pay metered costs for the diskspace used by the DB and actual CPU use by the customer.

I've looked at the various 'clouds' and VPS hosts and it seems most of these offerings are geared towards CPU- or traffic-heavy applications: I'd rather we dont pay for RAM or bandwidth we don't use. Amazon's cloud seems like the best fit, but I am confused as to how (if?) we could host a regular Linux app that reads access to a regular file-based DB which must persist. And my understanding of AWS is that they will charge for the instance as long as it's non-dormant, regardless if the customers are using it or not, correct?

I am looking for pointers/advice or experiences (good/bad) for similar situations...

Answer 1

I think that AWS is probably going to be one of your better options. Yes, as long as the VM is running you'll be charged for it, but you can run a small instance if you don't have heavy CPU and RAM requirements and it won't cost you too much.

As far as storing a persistent database, yes you can do this with EBS (Elastic Block Store), and you'll just get charged for the storage cost (which isn't much) if you're not actually doing any IO ops on it.

Answer 2

To squeeze out a lot of performance you will need to do a software RAID on EBS volumes, preferably RAID10, as well as tweak your MySQL.

There is no cloud which will give you granular control over the CPU and RAM, you can only buy an instance with a specific amount of RAM and CPU allocation. You can however scale these up, though some downtime may be required while the switch is performed.

If you're looking for control, you may as well get your own server in your office and let them play around, as this may give you better control as well as lower your costs.

There is also the option of renting a dedicated server and placing your customers there. Using a VPS will most likely not work, depending on your understanding of "I/O heavy" as in clouds and virtualized environments you are always sharing disk i/o with someone else, and may have peaks and lows when it comes to requests per second.