Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have 1 source server that contains about 30GB of files that I want to copy to 7 other servers. I currently SCP the files over to the first four servers and when that transfer is complete SCP to the last 3 servers.

Would using BitTorrent be faster to distribute the files to all 7 at once? The connection internally is already fast and consistent between each of the servers. I know it would probably take some load off the source server, but would it just take longer to distribute the files?

What's a good scriptable client I could use from the shell? I'd want everyone to stop seeding after 100%.

share|improve this question
add comment

migrated from stackoverflow.com Jun 2 '11 at 0:49

This question came from our site for professional and enthusiast programmers.

4 Answers

up vote 7 down vote accepted

It depends on how your network is laid out and where the bottlenecks are, but yes, it can be faster to use BitTorrent. In theory, instead of sending out 7 copies of the same files, your source server should only need to send out 1 copy, split among the receivers. By the time that happens, the remaining peers will have shared the parts they didn't receive directly among themselves.

If your network supports multicast, though, it's probably easier to set up udpcast, mdp, multisend, or something like that: your source will send out a single copy of the data, and the network will route it to all the recipients.

share|improve this answer
    
We ended up going the multicast method using Udpcast. And it works perfectly! –  Edward Dec 14 '11 at 1:09
add comment

The advantage of BitTorrent over other file distribution tools, say rsync, is that clients start to offer their data to the other clients, so if the server has a slow connection to the clients, throughput will be much better.

BT is a protocol which is designed for global sharing, though. So you will have to find a means to lock out all other clients but the 7 you want to feed (or your torrent will soon end up on The Pirate Bay).

As for stopping at 100%, every BT client will do that: Since no client requests the data anymore, there will be no more seeding. The application will still run but they will only cause a little trickle of data between the clients and the tracker to tell it that they are still available.

Try the original BitTorrent client which is written in Python. This will allow you to make any modifications you desire. Unfortunately, I can't find a link ATM. Anyone?

[EDIT] Based on a comment saua: If you use a private tracker, your torrents should not spread unexpectedly. That said, you should only use a torrent client from a reliable source. Otherwise, you might risk that some cracker has put a little extra feature in the code which publishes private torrents somewhere where you don't want them to appear.

share|improve this answer
2  
Not making the torrent public is easily done by using a private tracker. –  Joachim Sauer Feb 3 '09 at 15:58
1  
Are you 100% sure that the source code of the client doesn't contain a "hey, let's tell some hacker server about this torrent" piece of code? I'd rather not risk my neck here. –  Aaron Digulla Feb 3 '09 at 16:50
7  
If you don't trust a piece of code and run it anyway, having it leak data about your private torrents is the least of your worries. –  Michael Borgwardt Jun 22 '09 at 21:58
add comment

BitTorrent is a great idea if you want to replicate files through a tree of servers without for example having to issue a remote command to all servers to copy from the same place; potentially causing a bottleneck.

Twitter engineers done this with great success with Murder, which is now on github.

http://engineering.twitter.com/2010/07/murder-fast-datacenter-code-deploys.html

share|improve this answer
add comment

If all 8 of your servers are plugged into the same section of the network using Bit Torrent is unlikely to speed things up. In fact you would get so many collisions that it would probably be very slow. BitTorrent would only speed things up if you are using a fully switched network and the eight servers can talk to each other independently without affecting the communication between the other servers.

share|improve this answer
    
Good point - but are there really many networks out there that aren't fully switched anymore? –  phyllis diller Mar 19 '10 at 19:25
2  
When was the last time when you have seen a hub connecting servers? These days all of us are using full duplex switched networks. –  Mircea Vutcovici Jun 2 '11 at 4:56
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.