realtime sync of files between instances

Question

In my scenario, i have 2 ec2 instances and need to sync the files between them in realtime. Can I do this with rsync? Can anyone please suggest how to schedule a rsync?

Answer 1

Don't try to sync file systems. Read and write your files to S3 directly.

Syncing file systems adds an additional level of complexity to your application that's unnecessary. If the synchronization of the file system fails for any reason, your app will start failing.

Keep things simple: construct your application to use S3 effectively.

Answer 2

You have many alternatives:

1. You can use Amazon's S3 to store files and access it via s3fs on both machines.
2. You can use some cluster filesystem like GlusterFS or GVFS2.

Answer 3

If Linux, I would suggest using NFS filesystems/mounts across the nodes that require access to the files. Have you considered this?

Answer 4

You can use some file sync tool like rsync or unison plus incrontab.

Answer 5

If your instances do concurrent write access to the files in question, consider using a distributed file system capable of propagating locks instead. It would not need scheduling and you would not need to worry about update conflicts (i.e. a situation where a file has been updated on both instances before synchronization).

GFS2 might or might not fit your needs - check it out.

Answer 6

Using single file system is bad singe you are introducing single point of failure in your system. Making good architecture design from start is key to scaling your application. Depending on local disk storage is never good option. If you are writing system that will be working across number of nodes you should consider using some TCP service which any node can access over network.

On the other hand if you have service on multiple systems and those systems are going to be backends behind some load balancer, given the assumption that files you need to have shared are related exclusively to visitors of given backend, you could try with stickiness so every returning visitor always hits same backend.

If however none of these work for you, cross syncing local files to multiple servers will be hard problem to resolve so as ewwhite suggested, NFS file storage looks like only sane option then.