I'm attempting to copy a 75 gigabyte tgz (mysql lvm snapshot) from a Linux server in our LA data center to another Linux server in our NY data center over a 10MB link.
I am getting about 20-30Kb/s with rsync or scp which fluctates between 200-300 hours.
At the moment it is a relatively quiet link as the second data center is not yet active and I have gotten excellent speeds from small file transfers.
I've followed different tcp tuning guides I've found via google to no avail (maybe I'm reading the wrong guides, got a good one?).
I've seen the tar+netcat tunnel tip, but my understanding is that it is only good for LOTS of small files an doesn't update you when the file is effectively finished transferring.
Before I resort to shipping a hard drive, does anyone have any good input?
UPDATE: Well... it may be the link afterall :( See my tests below...
Transfers from NY to LA:
Getting a blank file.
[nathan@laobnas test]$ dd if=/dev/zero of=FROM_LA_TEST bs=1k count=4700000 4700000+0 records in 4700000+0 records out 4812800000 bytes (4.8 GB) copied, 29.412 seconds, 164 MB/s [nathan@laobnas test]$ scp -C obnas:/obbkup/test/FROM_NY_TEST . FROM_NY_TEST 3% 146MB 9.4MB/s 07:52 ETA
Getting the snapshot tarball.
[nathan@obnas db_backup]$ ls -la db_dump.08120922.tar.gz -rw-r--r-- 1 root root 30428904033 Aug 12 22:42 db_dump.08120922.tar.gz [nathan@laobnas test]$ scp -C obnas:/obbkup/db_backup/db_dump.08120922.tar.gz . db_dump.08120922.tar.gz 0% 56MB 574.3KB/s 14:20:40 ET
Transfers from LA to NY:
Getting a blank file.
[nathan@obnas test]$ dd if=/dev/zero of=FROM_NY_TEST bs=1k count=4700000 4700000+0 records in 4700000+0 records out 4812800000 bytes (4.8 GB) copied, 29.2501 seconds, 165 MB/s [nathan@obnas test]$ scp -C laobnas:/obbkup/test/FROM_LA_TEST . FROM_LA_TEST 0% 6008KB 497.1KB/s 2:37:22 ETA
Gettting the snapshot tarball.
[nathan@laobnas db_backup]$ ls -la db_dump_08120901.tar.gz -rw-r--r-- 1 root root 31090827509 Aug 12 21:21 db_dump_08120901.tar.gz [nathan@obnas test]$ scp -C laobnas:/obbkup/db_backup/db_dump_08120901.tar.gz . db_dump_08120901.tar.gz 0% 324KB 26.8KB/s 314:11:38 ETA
I guess I'll take it up with the folks who run our facilities the link is labeled as a MPLS/Ethernet 10MB link. (shrug)