Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Im currently building a small CDN topology system, and im looking for the best way to sync the files within certain folder in all the servers.

I need to be able to upload a file to a certain SINGLE FTP, and to get it to duplicate to all the other servers.

The size of the folder that needs to be synced is around 50GB, and i need it to be synced in Up-To 5 minutes difference.

The O/S on all servers is windows 2008 R2.

Any ideas ?

share|improve this question

migrated from stackoverflow.com Jul 22 '11 at 4:08

This question came from our site for professional and enthusiast programmers.

    
Is this a programming question or are you looking for a software that will do this for you? –  Shaded Jul 20 '11 at 15:02
    
Software. I have several Non-Developers end users that i need them to upload one time and get it duplicated onto several other servers –  DeaglinG Jul 21 '11 at 16:30
    
Then this would probably be better posted on Server Fault, since you're not looking to develop anything. –  Shaded Jul 21 '11 at 16:36

3 Answers 3

up vote 1 down vote accepted

I have eventually used a Enterprise-Grade program called RepliWeb, which is doing a terrific job!

Well recommended!

share|improve this answer
    
Glad you resolved this issue. Please mark your answer as "accepted" when you're able. –  jscott Dec 6 '12 at 17:48

On Linux I'd suggest using RSYNC, there are a couple of RSYNC 'clones' for Windows

RSYNC is great for transfering files around, you can tell it to only transfer new or modified files and even delete old files not avaliable on the origin server. I'm by no means a Windows expert, but I would guess you could set this up in a PowerShell/Batch script and run it ether automatically every few minutes or manually when you need the update. If there was nothing new to transfer RSYNC would figure it out and not run so you wouldn't be using bandwidth.

The second option, depending on your skillset etc. is to develop some custom software to do this. I would guess it would be relatively simple, you could do it in something like Python, Ruby or even PHP but it would need to follow a few basic steps;

  1. Load a list of remote servers
  2. Get a list of local files and their last-modified date
  3. Connect to one* remote server using FTP or something similar, list the files and get the last-modified date on the remote server
  4. Figure out which files on the remote servers need to be synced (The last-modified date will be different, or the files won't exist)
  5. Transfer the files to all the remote servers

*I say one server, assuming all the servers will be in sync I.E. the file mod times will be the same

I remember building a similar script to this in Python, when I get home later I'll see if I can dig it up for you, but I'd still suggest RSYNC is the best option if you can get it to work (I.E. it does all of this already)

share|improve this answer
    
Im looking for something a bit more secure. Relying on a script is kinda risky. –  DeaglinG Jul 25 '11 at 13:03

Solutions: Assuming that you truly have one master FTP and various slaves:

After uploading, trigger e.g by calling yourwebsite/distribute.php an system('distribute.sh') which contains:

rsync <a ton of parameters to upload to Slave A>
rsync <a ton of parameters to upload to Slave B>
rsync <a ton of parameters to upload to Slave C>

rsync does support lazy synchronization. So you won't keep pushing around the whole thing.

Another way should be (ab)using the versioning system 'git' by the way. So anything uploaded there gets auto-added to a depot (I think git uses a differnt term), your slaves frequently (e.g. cronjob) sync to that git. Benefit might be (more) transactional safety. No danger of distributing half-uploaded files to the slaves.

Perforce (afaik legally free for up to 2 users and 5 client specs. Very, very stable, also with huge binaries) or SVN (annoying .snv subfolders. And I wouldn't say this stable thing...) might do the same for you. Perforce IMO is easier to grasp than git.

But git might be the 'more modern' way to go, and free. Not sure, how it responds to huge binaries, i.e. if it can be talked out of trying to diff'em. Or keep versioning depth on slaves at zero...

Comment: Hey, I wonder if you are in the same shoes like me: I do press photography, so in the field I have to use an annoying, unstable, tiny upload connection to get to (ideally just 1) FTP server. From there (my webhosters server centre), it would be great if it could be fast distributed through those (I trust: arm-thick T1 cables...) to all the image agencies...

I haven't scripted a solution yet, but I am about to...

share|improve this answer
    
FYI, Cronjobs don't work on Windows, there is something similar though iopus.com/guides/winscheduler.htm - But +1 for GIT (although all the slaves would have to have their own scheduled tasks unless you set up hooks on the GIT server) –  sam Jul 22 '11 at 8:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.