Linking 2 EC2 Instances

Question

If I have one EC2 instance that is simply for storing files and another EC2 instance that serves a webpage with apache, how do I get them to "link" together?

Answer 1

The scenario you describe (one server running apache and another storing files) does not offer any real backup - arguably, it doubles the chance of failure (you now have two machines, and if either fail, the entire system will not work).

If you are just starting with EC2, the simplest approach to get similar functionality, would be to have your data stored on an EBS drive, and to have a your customized instance saved as an AMI. From there, you could setup autoscaling to maintain 1 instance at all times - if your one instance fails, another can be brought online based on your AMI, and you can either use a snapshot (slightly outdated), or move the EBS volume to get everything back up and running (which can be automated).

If you are just serving static files, then you can do away with EC2 altogether and simply use S3 - it will provide the scalability and (internal) redundancy that you desire. If however, you are using databases, then it is not a good idea to store those on S3 as they will incur a significant performance loss.

If you are really looking for a failover setup, you will want your setup to function even if an instance fails. With two instances, that would imply that either a) each is self-sufficient, or b) when one fails, another equivalent instance will be brought online.

The potential difference that exists between the failover setup described above, and an autoscaling cluster lies in the degree of control you have. While Cloudwatch (and thus autoscaling) does support custom metrics, other solutions allow more versatility in their implementation. For instance (and purely theoretical), you can setup a 2 node cluster (e.g. in Amazon's VPC) and use Heartbeat/Corosync to monitor the status of each instance, with Pacemaker managing resources (e.g. if your apache fails instead of the entire instance, some action can be taken). Files could be shared between the instances either with a simple rsync (if there isn't a lot of change), or with a distributed file system (e.g. Gluster). One instance, would be elected the 'leader' and can run a load balancer (e.g. nginx, HAProxy, etc) and send requests to one instance or the other. Such a design is not without its problems (and certainly takes a good bit of effort to setup), but may provide some idea of a potential architecture.

To answer the question as asked however (which essentially amounts to how to access remote files locally). You can either go with:

• A shared storage approach (files stored on S3, which is mounted locally using fuse)
• A network file system (e.g. Gluster - remote file system made accessible to networked instances)
• Remote file system accessed locally (e.g. sshfs - fuse based approach to accessing/mounting remote file system over ssh/sftp)

With almost any approach you choose, you will need to open ports on your security group to allow communication between the instances - you may want to restrict access to those ports to only other instances in the same security group.