9

I have a few large files that I need to copy from one Linux machine to about 20 other Linux machines, all on the same LAN as quickly as is feasible. What tools/methods would be best for copying these files, noting that this is not going to be a one-time copy. These machines will never be connected to the Internet, and security is not an issue.

Update:

The reason for my asking this is because (as I understand it) we are currently using scp in serial to copy the files to each of the machines and I have been informed that this is "too slow" and a faster alternative is being sought. According to what I have been told, attempting to parallelize the scp calls simply slows it down further due to hard drive seeks.

Improve this question
3
  • Define "large". Hundreds of MB/GB/TB/more?
    – Janne Pikkarainen
    Commented Nov 18, 2011 at 13:44
  • Currently, the total is sitting at about 4 GiB (compressed), although that may grow in the future.
    – Jonathan Callen
    Commented Nov 18, 2011 at 13:46
  • That is, by any means even in 2011 - NOT LARGE. Given a 1gigabit link that is properly switched (standard in 2011) that is quite fas to accomplish. Running it of a 10g server (not that rare even in 2011)... well ;)
    – TomTom
    Commented Mar 27, 2018 at 20:10

5 Answers 5

Reset to default
26

BitTorrent. It's how Twitter deploys some things internally.

http://engineering.twitter.com/2010/07/murder-fast-datacenter-code-deploys.html (web archive link)

Improve this answer
13
  • 3
    In addition to my answer (which I do think will do a good job, if you can implement it), the below answer for NFS is a very good one. A good NFS server should cache files so you won't keep hitting disk. Also, with that, don't copy the files from the server to the clients. Initiate it from the client and let the NFS server's cache help out.
    – mfinni
    Commented Nov 18, 2011 at 15:24
  • 1
    Be sure to try it in a non-production environnement, in the presentation they say (iirc) it made some switches suffer a lot during the first deployments because of the number of packets exchanged.
    – Shadok
    Commented Nov 18, 2011 at 16:55
  • 1
    @psusi Why do you say it has to send all of the data 20 times? Once the other peers have part of the file, they can start sending the parts they have to the other peers themselves.
    – Jonathan Callen
    Commented Nov 18, 2011 at 18:11
  • 2
    The problem for the OP is not the LAN, it's the disk on the central server.
    – mfinni
    Commented Nov 18, 2011 at 18:16
  • 1
    @pSusi - multicast would certainly be another valid answer. Post that as an answer, not as a knock on my answer.
    – mfinni
    Commented Nov 18, 2011 at 18:18
12

How about UFTP, it uses multicast to deliver files over UDP to multiple clients at once. Not for everyone and I'm no expert on it but it sounds like it does what you want.

Improve this answer
4
  • 1
    Disclaimer: This will require equipment that supports multicast.
    – user606723
    Commented Nov 18, 2011 at 16:33
  • I was rather hoping this'd be on the same vlan - reducing the impact of this use.
    – Chopper3
    Commented Nov 18, 2011 at 16:56
  • @user606723: Doesn't everything modern? Maybe some consumer junk does not, but I haven't run into anything with broken multicast in a while. Too much uses it these days. I think Windows Active Directory even uses multicast.
    – Zan Lynx
    Commented Nov 18, 2011 at 19:08
  • I actually have no experience with this @ZanLynx. I know that many offices/computer labs use consumer/unmanaged switches at the last hop. How will these switches behave with multicast?
    – user606723
    Commented Nov 18, 2011 at 19:42
3

Have you tried to copy this data with rsync? If you have 1 Gbit LAN or faster, copying over 4*20 GB should not be a problem.

How often will this copy occur? Does it matter if it takes couple of minutes to finish?

Improve this answer
3

scp-tsunami it's the way!

https://code.google.com/p/scp-tsunami/

It's commonly used to distribute disk images on virtualization clusters, its performances are near bittorrent but it's simpler to use for daily usage.

Improve this answer
2

Setting up an NFS share and having each machine pull from this shared repo of large files would likely be the fastest method (NFS is very quick and has little overhead).

You could add an additional NIC or two to the source server and bond them together to give you better throughput.

Implementation could be a simple cron job on each target server that blindly fetches from the share every hour/day/whatever. You could also setup a daemon to poll for new files; you could also just script a control session to SSH (with key pairs) into each target box and instruct them to fetch the file when you execute your script.

Improve this answer
1
  • 1
    I believe my predecessor attempted to use NFS for this and found that (at the time), the RAM cache wasn't large enough for the entire transfer, which was causing the load on the hard drive to become the limiting factor instead of the network speed.
    – Jonathan Callen
    Commented Nov 18, 2011 at 15:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .