Migrating to AWS Cloud with auto-scaling - where to put Redis and ElasticSearch?

Question

I've been trying to research this topic but haven't found anywhere that recommends where to install services such as Redis and ElasticSearch when migrating to a cloud framework.

I'm currently running a Symfony2 application on 2 static servers - one is running MySQL and the other is the public facing web server, which also has Redis and ElasticSearch running on it. Both of these servers are virtualised, but they're static in terms of not being able to replicate at present (various aspects are still dependent on the local filesystem).

The goal is to migrate to AWS and use auto-scaling to be able to spin up and kill web servers as required, but I'm not clear on what I should put on each EC2 instance. Should they be single-responsibility only? i.e. Set up individual instances for the web server(s), Redis, and ElasticSearch and most likely an RDS instance for MySQL and only set up auto-scaling on the web server(s)?

I don't foresee having to scale the ElasticSearch server anytime soon as it's only driving the search functionality, but it's possible that Redis may need to be replicated at some point - but should this be done manually? I'm not sure of how this could be done automatically as each instance needs to be configured to know about it's master/slave(s) as far as I know. I'd appreciate advice on this.

One more quick question while I'm here - how would I be able to deploy code changes when there are X web servers currently active? I'm using a Capifony deployment script (Symfony2 version of Capistrano), which I think can handle multiple servers easily enough by specifying an array of :domain addresses...but how can should this be handled when the number of web servers can vary?

Answer 1

The way we handle this is by creating multiple groups of servers in a layered stack (even if a group currently only needs one instance). The first layer is your Elastic Load Balancer, clearly.

The second layer is an Auto Scaling Group of web servers (multi-availability zone). These boot a custom AMI designed be in a proper ready-state for this task on startup. (Now that our processes are more mature, we actually boot a generic AMI that can auto-configure on startup using Chef.) But we also do a git pull of the latest production code repository on startup, so we don't have to create a new AMI with each code deployment. This also allows us to change configurations, such as database hosts, Redis hosts, etc more easily.

The third layer is for database and other services such as ElasticSearch and Redis. You can either host all three services on one box, then deal with managing your own mysql slaves, or you can host Redis and ElasticSearch on their own box and use Amazon's RDS for your Mysql services. Your choice, based on whether or not you want to manage your own replication/fault-tolerance in MySQL.

Often the simplest way is to use Amazon RDS in a multi-availability zone configuration. We always try to deploy multi-AZ with everything, so we are still up-and-running if a single AZ fails. Then you run a smaller instance to host just Redis and ElasticSearch.

With ElasticSearch, here's a tip we use for rails installs: Install and maintain a complete instance of your app along with ElasticSearch on the box. Then build an AMI for this role (or a Chef role). The reason is so that you can run the utility tasks on bootup to create your ElasticSearch indices from scratch if you're booting a fresh AMI. Then, put this instance in an multi-AZ ASG as well, with a min and max of one server. If that box or AZ dies, the ASG will boot up a replacement, and it will rebuild it's indices on startup and be ready to serve clients.

For Redis, there is good news on the horizon. redis-cluster is coming soon, which promises to allow for easier management of scaling redis stores. In the meantime, you can handle your own replication or try Garantia, a hosted scalable redis server solution, which is using a version of redis-cluster in beta now (currently limited to us-east-1 region). This has the advantage of keeping the same IP addresses for your configurations, no matter what happens to your instance pools.

Finally, to protect your data going to-and-from your databases, I would recommend building this inside the private network portion of a public/private Virtual Private Cloud. This sets up your own private network that is isolated from packet sniffers. You can also employ SSL encryption for your MySQL database connections.