Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am doing some distributed work with RackSpace cloud servers and I am using bittorrent to distribute my files. It works surprisingly well. However, distributing the torrent files themselves are not so nice. How would you go around doing that? Right now I just scp the torrent files to the servers, and of course I could write a script that copies it to sqrt(n) servers instructing each to again copy to sqrt(n) but that's a pita to work it.

share|improve this question

4 Answers 4

up vote 23 down vote accepted

Not knowing what exactly your problem is, I can recommend pscp from parallel-ssh as a tool to upload small files to multiple servers.

You prepare a list of servers to upload to and let it know what to take locally and where to put it remotely. For example:

$ pscp -h list-of-servers file.torrent /tmp/
[1] 02:11:22 [SUCCESS] 10.0.0.21
[2] 02:11:22 [SUCCESS] 10.0.0.20
[3] 02:11:22 [SUCCESS] 10.0.0.45
[4] 02:11:22 [SUCCESS] 10.0.0.19
[5] 02:11:22 [SUCCESS] 10.0.0.2
[6] 02:11:22 [SUCCESS] 10.0.0.5
[7] 02:11:25 [FAILURE] 10.0.0.3 Exited with error code 1
share|improve this answer
1  
There's also code.google.com/p/pdsh –  chx Oct 31 '12 at 20:03

Given they're very small files typically couldn't you just have a 'dropbox' using HTTP with a cron'ed script doing a curl/wget with wildcard?

share|improve this answer
1  
This is the method I would use, no need to push a file out to each server and maintain a server list, have them periodically check for new .torrent file –  jwbensley Oct 31 '12 at 17:41
1  
you could also use Puppet to do this –  Noah Yetter Nov 1 '12 at 5:20

You could have the torrent clients check an rss feed for new torrent files or switch to magnet links instead of torrent files.

share|improve this answer
    
RSS may not be an option if he/she is using a basic command line torrent client. –  Hengjie Oct 31 '12 at 22:44
    
He could you Flexget with rtorrent without gui –  chewbakka Nov 1 '12 at 11:07
    
Yep, flexget and rtorrent would work well for this –  Grant Nov 2 '12 at 13:09
    
+1 "switch to magnet links instead of torrent files." –  michael_n Jan 11 at 12:14

You might want to look into murder

Murder is a method of using Bittorrent to distribute files to a large amount of servers within a production environment. This allows for scaleable and fast deploys in environments of hundreds to tens of thousands of servers where centralized distribution systems wouldn't otherwise function

.

Murder was/is developed by the folks at twitter, and they're using it daily to distribute files.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.