13

I have 1 source server that contains about 30GB of files that I want to copy to 7 other servers. I currently SCP the files over to the first four servers and when that transfer is complete SCP to the last 3 servers.

Would using BitTorrent be faster to distribute the files to all 7 at once? The connection internally is already fast and consistent between each of the servers. I know it would probably take some load off the source server, but would it just take longer to distribute the files?

What's a good scriptable client I could use from the shell? I'd want everyone to stop seeding after 100%.

Improve this question

4 Answers 4

Reset to default
9

It depends on how your network is laid out and where the bottlenecks are, but yes, it can be faster to use BitTorrent. In theory, instead of sending out 7 copies of the same files, your source server should only need to send out 1 copy, split among the receivers. By the time that happens, the remaining peers will have shared the parts they didn't receive directly among themselves.

If your network supports multicast, though, it's probably easier to set up udpcast, mdp, multisend, or something like that: your source will send out a single copy of the data, and the network will route it to all the recipients.

Improve this answer
1
  • We ended up going the multicast method using Udpcast. And it works perfectly!
    – Edward
    Dec 14, 2011 at 1:09
3

The advantage of BitTorrent over other file distribution tools, say rsync, is that clients start to offer their data to the other clients, so if the server has a slow connection to the clients, throughput will be much better.

BT is a protocol which is designed for global sharing, though. So you will have to find a means to lock out all other clients but the 7 you want to feed (or your torrent will soon end up on The Pirate Bay).

As for stopping at 100%, every BT client will do that: Since no client requests the data anymore, there will be no more seeding. The application will still run but they will only cause a little trickle of data between the clients and the tracker to tell it that they are still available.

Try the original BitTorrent client which is written in Python. This will allow you to make any modifications you desire. Unfortunately, I can't find a link ATM. Anyone?

[EDIT] Based on a comment saua: If you use a private tracker, your torrents should not spread unexpectedly. That said, you should only use a torrent client from a reliable source. Otherwise, you might risk that some cracker has put a little extra feature in the code which publishes private torrents somewhere where you don't want them to appear.

Improve this answer
3
  • 2
    Not making the torrent public is easily done by using a private tracker.
    – Joachim Sauer
    Feb 3, 2009 at 15:58
  • 1
    Are you 100% sure that the source code of the client doesn't contain a "hey, let's tell some hacker server about this torrent" piece of code? I'd rather not risk my neck here.
    – Aaron Digulla
    Feb 3, 2009 at 16:50
  • 7
    If you don't trust a piece of code and run it anyway, having it leak data about your private torrents is the least of your worries.
    – Michael Borgwardt
    Jun 22, 2009 at 21:58
1

BitTorrent is a great idea if you want to replicate files through a tree of servers without for example having to issue a remote command to all servers to copy from the same place; potentially causing a bottleneck.

Twitter engineers done this with great success with Murder, which is now on github.

http://engineering.twitter.com/2010/07/murder-fast-datacenter-code-deploys.html

Improve this answer
0

If all 8 of your servers are plugged into the same section of the network using Bit Torrent is unlikely to speed things up. In fact you would get so many collisions that it would probably be very slow. BitTorrent would only speed things up if you are using a fully switched network and the eight servers can talk to each other independently without affecting the communication between the other servers.

Improve this answer
2
  • Good point - but are there really many networks out there that aren't fully switched anymore?
    – phyllis diller
    Mar 19, 2010 at 19:25
  • 2
    When was the last time when you have seen a hub connecting servers? These days all of us are using full duplex switched networks.
    – Mircea Vutcovici
    Jun 2, 2011 at 4:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .