Uploading to multiple FTP sites at once

Question

Im currently building a small CDN topology system, and im looking for the best way to sync the files within certain folder in all the servers.

I need to be able to upload a file to a certain SINGLE FTP, and to get it to duplicate to all the other servers.

The size of the folder that needs to be synced is around 50GB, and i need it to be synced in Up-To 5 minutes difference.

The O/S on all servers is windows 2008 R2.

Any ideas ?

Answer 1

I have eventually used a Enterprise-Grade program called RepliWeb, which is doing a terrific job!

Well recommended!

Answer 2

On Linux I'd suggest using RSYNC, there are a couple of RSYNC 'clones' for Windows

RSYNC is great for transfering files around, you can tell it to only transfer new or modified files and even delete old files not avaliable on the origin server. I'm by no means a Windows expert, but I would guess you could set this up in a PowerShell/Batch script and run it ether automatically every few minutes or manually when you need the update. If there was nothing new to transfer RSYNC would figure it out and not run so you wouldn't be using bandwidth.

The second option, depending on your skillset etc. is to develop some custom software to do this. I would guess it would be relatively simple, you could do it in something like Python, Ruby or even PHP but it would need to follow a few basic steps;

1. Load a list of remote servers
2. Get a list of local files and their last-modified date
3. Connect to one* remote server using FTP or something similar, list the files and get the last-modified date on the remote server
4. Figure out which files on the remote servers need to be synced (The last-modified date will be different, or the files won't exist)
5. Transfer the files to all the remote servers

*I say one server, assuming all the servers will be in sync I.E. the file mod times will be the same

I remember building a similar script to this in Python, when I get home later I'll see if I can dig it up for you, but I'd still suggest RSYNC is the best option if you can get it to work (I.E. it does all of this already)

Answer 3

Solutions: Assuming that you truly have one master FTP and various slaves:

After uploading, trigger e.g by calling yourwebsite/distribute.php an system('distribute.sh') which contains:

rsync <a ton of parameters to upload to Slave A>
rsync <a ton of parameters to upload to Slave B>
rsync <a ton of parameters to upload to Slave C>

rsync does support lazy synchronization. So you won't keep pushing around the whole thing.

Another way should be (ab)using the versioning system 'git' by the way. So anything uploaded there gets auto-added to a depot (I think git uses a differnt term), your slaves frequently (e.g. cronjob) sync to that git. Benefit might be (more) transactional safety. No danger of distributing half-uploaded files to the slaves.

Perforce (afaik legally free for up to 2 users and 5 client specs. Very, very stable, also with huge binaries) or SVN (annoying .snv subfolders. And I wouldn't say this stable thing...) might do the same for you. Perforce IMO is easier to grasp than git.

But git might be the 'more modern' way to go, and free. Not sure, how it responds to huge binaries, i.e. if it can be talked out of trying to diff'em. Or keep versioning depth on slaves at zero...

Comment: Hey, I wonder if you are in the same shoes like me: I do press photography, so in the field I have to use an annoying, unstable, tiny upload connection to get to (ideally just 1) FTP server. From there (my webhosters server centre), it would be great if it could be fast distributed through those (I trust: arm-thick T1 cables...) to all the image agencies...

I haven't scripted a solution yet, but I am about to...