I need the ability to push images out to multiple servers whenever they're updated. I've looked at Network Filesystems, but they're all but worthless due to their speed. Images can be uploaded to any one of 3 servers, and would then need to be copied to the other 2.

Any suggestions? I'm open to try just about anything.

EDIT: Graphics data (jpg, gif, png, etc). Linux only.

We're currently using rsync. But having it work back and forth is getting cumbersome. It's all local network.

  • 2
    what os is this running on? – tony roth Dec 7 '10 at 19:48
  • 2
    When you say that network filesystems are worthless due to their speed, do you mean due to overhead? What have you looked at? Some network filesystems are pretty speedy. We have GPFS in production here, and while it's got its own share of nightmares, it's quite fast. (Faster than a local RAID 10.) – mattdm Dec 7 '10 at 19:55
  • 1
    By "images" do you mean graphics files or disk images? – mattdm Dec 7 '10 at 19:56
  • Are you saying "images" as in "data files that are graphics", or as in "disk dumps of the OS and applications" ? – mfinni Dec 7 '10 at 19:57

What OS? Are you transferring them across WAN links or within a data center? Absent this information, I'm going to give you a Linux/Unix answer.

rsync is good for handling file transfers of this sort, since it only transfers updates (rather than the entire file) when a file is modified. If you want this to happen automatically when a new image is uploaded, you probably want to modify your web application to kick off the appropriate process.

If you're working in a Windows environment there are Windows versions of rsync available.

  • Rsync isn't going to be particularly fast if the images are all-new, or if computing the difference happens to be more work than sending the whole file. Need... more... information! – mattdm Dec 7 '10 at 19:56
  • @mattdm: For GNU rsync the '-W' disables the incremental algorithm and just sends the whole file. – Scott Pack Dec 7 '10 at 20:03
  • @packs: sure. But then we're back to "why is a network filesystem considered slow?" – mattdm Dec 7 '10 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.