Tag Info

Hot answers tagged

164

Ship hard drives across the ocean instead. At 11 Mbps with full utilization, you're looking at just shy of 90 days to transfer 10 TB. 11 Mbps = 1.375 MBps = 116.015 GB/day. 10240 GB / 116.015 GB/day = ~88.3 days.


73

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag - archive. So: rsync -a source dest The default cp ...


65

I would recommend tar. When the file trees are already similar, rsync performs very well. However, since rsync will do multiple analysis passes on each file, and then copy the changes, it is much slower than tar for the initial copy. This command will likely do what you want. It will copy the files between the machines, as well as preserve both ...


52

I'd be tempted to rsync it over myself - it does compression and handles link loss well.


40

When I have to copy a large amount of data, I usually use a combination of tar and rsync. The first pass is to tar it, something like this: # (cd /src; tar cf - .) | (cd /dst; tar xpf -) Usually with a large amount of files, there will be some that tar can't handle for whatever reason. Or maybe the process will get interrupted, or if it is a filesystem ...


38

See Robocopy /? Usage : ROBOCOPY source destination [file [file]...] [options] robocopy c:\folder d:\folder transfer_this.txt


36

Instead of using tar to write to your local disk, you can write directly to the remote server over the network using ssh. server1$ tar -zc ./path | ssh server2 "cat > ~/file.tar.gz" Any string that follows your "ssh" command will be run on the remote server instead of the interactive logon. You can pipe input/output to and from those remote commands ...


31

There is a max-size option: --max-size=SIZE don't transfer any file larger than SIZE So: # rsync -rv --max-size=1.5m root@tss01:/tmp/dm Will send only files less than 1.5m. Regarding sizes from man: The suffixes are as follows: "K" (or "KiB") is a kibibyte (1024), "M" (or "MiB") is a mebibyte (1024*1024), and "G" (or "GiB") is a ...


24

I'd say rsync, at 11 MB/s you will look at 10-14 days and even if you get interrupted, rsync will easily start where it stopped last time. At 11 Mbps I'd ship the hard disks like suggested above :)


23

rsync I'd use rsync before I used ftp or tftp. More options and (in my experience) more reliable transfer.


21

To do file transfer over ssh you can use scp scp -r /srcdir/ user@remotehost:/destdir/ use rsync over ssh (see the -e parameter) rsync -e ssh -a /srcdir/ user@remotehost:/destdir/ use some tool that transfers data via stdin/out (tar, cpio, etc) cd /sourcedir; tar -c . | ssh username@remotehost bash 'cd /dstdir; tar -x Mount the filesystem via sshfs ...


20

Buy an external hard drive. Copy data at site A. Mail it to site B. Copy to machine at site B. Don't underestimate the power of the postal service.


19

tar over ssh is okay, but tar over TCP via netcat is about as low-overhead as you can get! If this is a one-time thing, give this a shot: On the receiver: nc -l -p 8989 | tar x On the sender: tar cf - /source-path | nc (receiving host ip address) 8989 If this is something you're going to do regularly, I'd probably use rsync.


18

External hard drive and same-day courier delivery.


18

You could use rsync(1): rsync --remove-source-files /path/to/source /path/to/destination This will remove successfully transferred files from the original path.


16

Using --link-dest to create space-efficient snapshot based backups, whereby you appear to have multiple complete copies of the backedup data (one for each backup run) but files that don't change between runs are hard-linked instead of creating new copies saving space. (actually, I still use the rysnc-followed-by-cp -al method which achieves the same thing, ...


16

The "Binary" transfer mode of FTP copies files exactly, byte for byte. Simple and straightforward. When bringing text files between different operating systems, though, this might not be what you want -- different operating systems use different codes to represent line breaks. The "ASCII" mode exists for this purpose: it automatically translates all line ...


16

Ugh. Your teacher is not correct at all. HTTP transfers files - that's how it works. It doesn't require anything special to do so, even if an Apache directory listing kinda looks like an FTP server listing. Look at your browser URL. Use a tool like Wireshark or Fiddler to actually look at the traffic. You'll see that if you're browsing via HTTP, and the ...


15

A failure predictable at 2Gb sounds like the target filesystem is to blame... Are both on NTFS? Are you piping through any compression (zip used to fail at 2gb boundaries) ((is apache doing compression)) I have copied many files over 20Gb using robocopy (as others have mentioned) but I'd avoid using the /MIR switch until you are sure you have got the copy ...


15

(answering my own question here :) ) The --remove-source-files will do this. :)


15

Try to use rsync version 3 if you have to sync many files! V3 builds its file list incrementally and is much faster and uses less memory than version 2. Depending on your platform this can make quite a difference. On OSX version 2.6.3 would take more than one hour or crash trying to build an index of 5 million files while the version 3.0.2 I compiled ...


14

Those large companies have geographically-dispersed data centres, so you're being responded to by a site that's closer to you rather than one central site - that's all it is.


13

I'd use rsync. If you've got them exported via HTTP with directory listings available, you could use wget and the --mirror argument, too. You're already seeing that HTTP is faster than SCP because SCP is encrypting everything (and thus bottlenecking on the CPU). HTTP and rsync are going to move faster because they're not encrypting. Here's some docs on ...


13

When copying to the local file system I always use the following rsync options: # rsync -avhW --no-compress --progress /src/ /dst/ Here's my reasoning: -a is for archive, which preserves ownership, permissions etc. -v is for verbose, so I can see what's happening (optional) -h is for human-readable, so the transfer rate and file sizes are easier to read ...


13

robocopy.exe has a switch called inter-packet gap, allowing you to insert a time window in between the packets of your copy, and thereby reduce the impact on the channel. It's not exactly "use no more than 30% of the available bandwidth", but you can acheive the same effect with a little math. You can always specify some number of milliseconds and let it ...


13

Searching for "high latency file transfer" brings up a lot of interesting hits. Clearly, this is a problem that both the CompSci community and the commercial community has put thougth into. A few commercial offerings that appear to fit the bill: FileCatalyst has products that can stream data over high-latency networks either using UDP or multiple TCP ...


13

Rsync of course. At least you can continue at any time after break , and its without any pain.


12

If you need to update a website with some huge files over a slowish link, you can transfer the small files this way: rsync -a --max-size=100K /var/www/ there:/var/www/ then do this for the big files: rsync -a --min-size=100K --bwlimit=100 /var/www/ there:/var/www/ rsync has lots of options that are handy for websites. Unfortunately, it does not have a ...


11

Rsync over SSH.


11

If you really want to use rsync, it sounds like you'll need some combination of --backup, --backup-dir, and --suffix. The closest I think you could get is with something like this rsync -abv --suffix R1 --remove-source-files src/ dst/ This would do close to what you want, but it would not rename the files exactly the way you'd want. The --suffix option ...



Only top voted, non community-wiki answers of a minimum length are eligible