Best practice for storing read-only instance data?

Question

So, i'm currently writing my bachelor thesis and my job is to cloudify a webservice which recommends a soundtrack for images. The main part of the recommendation process is a lookup on a ~40gb index file. The index file is read only and reading it should be as fast as possible. I also want to automatically launch additional instances based on demand. I did some research and have 3 possible ways to do it

1. Increase the root partition of the ec2 instance (to ~50gb), store the index file on the root partition and create an AMI. The benefit of this approach is, that it is very easy to start a new instance since everything is included in the AMI. But i also read that the speed of the root partition is very slow.
2. Store the data on a EBS volume, create a snapshot of it and whenever i launch a new instance, i create a new EBS Volume from the snapshot an attach it to the instance. I suppose this is the best way to do it but launching a new instance is a tad more complex
3. Storing the index file on S3 and whenever a new ec2 instance is launched, i download the file to the ephemeral storage of the instance. Problem with that approach is that it takes more time until the new instance is operable as well as the the cost for traffic.

I was also thinking about doing a Raid0 setup to improve the read performance on the index file. So which approach should i use?

Answer 1

Is this webservice already existing or something you're making or...? Because my answer would be to not cloudify it at all. I'd get a decent server with SSD's in it for maximum read performance.

But you'd also have to figure out what you're getting from the "cloud" side. "Clouds" just allow you to create many instances in a hopefully geographically diverse set of data centers; even then you don't guarantee that your end users will get good response time (like with Amazon, how do you get consistent performance if someone hits your server in a datacenter in New York and your client is in Australia?)

Where are your performance bottlenecks and how do you decouple elements to increase performance? Disk reads themselves would be taken care of with SSD's. "Clouds" don't magically increase performance; it's very much a function of architecture of the application. I haven't tested it and would like to know general numbers if someone has, but the proposal of launching various instances on demand when you are looking for increased performance seems like it would incur a greater overhead that would kill your database performance.

Also, you're focusing on disk when you might want to take a look at caching the @#% out of it. No matter how fast your disk performance you're not going to outstrip a good set of caching servers to keep the records hot in memory versus cold on disk. Again, a function of architecture of the application. And that's another thing that can hurt you in launching more VM's; spinning up VM's can kill caches and cause a lag before the caches are "primed", so to speak.

Answer 2

Comment numbers correspond to your options above.

1. I always recommend putting data on a separate EBS volume from the root, but if it's read only and easy to obtain when you are building AMIs, then I might make an exception. I don't see why an EBS root volume should be any slower than any other EBS volume.

2. This is easy if you set up your AMI correctly. See options for setting up block devices in ec2-register. Amazon can automatically create volumes for you based on snapshots and attatch them to the new instance, just like they do for the root volume.

3. There is no charge for traffic between EC2 instances and S3 in the endpoint related to the region. There are charges for EBS volumes and IO on EBS volumes.

Just because an EBS volume created from a snapshot is ready to use before a full download completes of a big file from S3, doesn't necessarily mean that EBS is faster. The volume is ready to accept operations almost immediately, but you will experience high iowait while the blocks populate from the snapshot.

Depending on the performance requirements of your application, you might find you need to "warm up" the EBS volume before you can put it into production. This is effectively the same thing as downloading it from S3. (I'd love to see some performance tests of these options.)

This all assumes that getting the index on disk is the right thing to do. For the best performance you might try loading it all into memory if you can afford instances big enough. On the other hand, simply using a high memory instance with the index on disk might eventually load it into memory anyway using standard disk buffering in the OS.

Depending on what you're indexing and how you access it, you might also take a look at SimpleDB, RDS, ElastiCache.