Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

In my scenario, i have 2 ec2 instances and need to sync the files between them in realtime. Can I do this with rsync? Can anyone please suggest how to schedule a rsync?

share|improve this question
    
It depends on what exactly you mean by 'in real time'. –  reinierpost Oct 5 '12 at 11:55
add comment

6 Answers 6

Don't try to sync file systems. Read and write your files to S3 directly.

Syncing file systems adds an additional level of complexity to your application that's unnecessary. If the synchronization of the file system fails for any reason, your app will start failing.

Keep things simple: construct your application to use S3 effectively.

share|improve this answer
add comment

You have many alternatives:

  1. You can use Amazon's S3 to store files and access it via s3fs on both machines.
  2. You can use some cluster filesystem like GlusterFS or GVFS2.
share|improve this answer
add comment

If Linux, I would suggest using NFS filesystems/mounts across the nodes that require access to the files. Have you considered this?

share|improve this answer
add comment

You can use some file sync tool like rsync or unison plus incrontab.

share|improve this answer
add comment

If your instances do concurrent write access to the files in question, consider using a distributed file system capable of propagating locks instead. It would not need scheduling and you would not need to worry about update conflicts (i.e. a situation where a file has been updated on both instances before synchronization).

GFS2 might or might not fit your needs - check it out.

share|improve this answer
add comment

Using single file system is bad singe you are introducing single point of failure in your system. Making good architecture design from start is key to scaling your application. Depending on local disk storage is never good option. If you are writing system that will be working across number of nodes you should consider using some TCP service which any node can access over network.

On the other hand if you have service on multiple systems and those systems are going to be backends behind some load balancer, given the assumption that files you need to have shared are related exclusively to visitors of given backend, you could try with stickiness so every returning visitor always hits same backend.

If however none of these work for you, cross syncing local files to multiple servers will be hard problem to resolve so as ewwhite suggested, NFS file storage looks like only sane option then.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.