I have a Window Server 2000 machine running MS SQL Server that stores over 20GB of data. The database is backed-up every day to the second harddrive. I want to transfer those backup files to another computer to build another test server and for recovery practicing. (the backup never actually got restored for almost 5 years. Don't tell my boss about that!)
I have trouble transfering that huge file through the network. I've tried plain network copy, apache download, and ftp. Any method I tried end up failing when the amount of data transfered reach 2GB. The last time that I successfully transfered the file, it was through a usb attached external harddrive. But I want to perform this task routinely and preferably automatically.
Wonder what is the most pragmatic approach for this situation ?
A failure predictable at 2Gb sounds like the target filesystem is to blame... Are both on NTFS? Are you piping through any compression (zip used to fail at 2gb boundaries) ((is apache doing compression))
I have copied many files over 20Gb using robocopy (as others have mentioned) but I'd avoid using the /MIR switch until you are sure you have got the copy doing what you want - since it will delete files as well as copy them.
SMB suffers from a one packet at a time limit so is often the slower way to copy files - you have the option to copy using push or pull. Personally, I prefer the push method (copy is initiated by the source).
As far as file copy utilities go, TeraCopy is a nice GUI-based one (not command line) that can queue lots of files, supports pausing and resuming, can dynamically change its buffer size to optimize speed, and can optionally replace Windows Explorer's default copy/move with its own.
The most pragmatic solution to repeated shuffles of large SQL Server backup files is to use a third party backup compression product or SQL Server 2008 Enterprise Edition's built-in backup compression.
There's several out there from different vendors. I work for Quest Software, the makers of LiteSpeed, but I'm not here to sell anything. You want to check out all of the products out there and decide what's best for your needs. Here's a recent blog post talking about LiteSpeed specifically, but the same concepts apply to other products too:
Its a bit late but I would recommend a 3rd party backup and restore application option. We use Red Gate SQL Backup (www.red-gate.com), it has compression and alternate location options in the GUI. I get compressions saving of 80% average - so you only transfer 20% of the actual db size. It also supports encryption so it can be used over a WAN without interception worries.
Its fully schedulable so can run automatically at a cycle of your choosing.
The GUI also allows you to configure and administer log shipping.
I don't have experience with such a large file, but could you use robocopy or even xcopy with the /Z option that claims to be restartable. It seems like this is intended for large file copies where the network is not reliable.
Use Netcat. A unix oriented tutorial for transferring files can be found here. You can further speed things up by:
Compress on the sender side and decompress on the destination side. (chuck windows equivalent of gzip in the middle of the command lines.)
Choose to send data via udp instead of by tcp (hey man, who cares about data integrity??!!)
Joking aside, netcat is likely the quickest way to transfer large files on a LAN. As no checksum is done you may wish to do a MD5 sum of the file before sending it and compare it to the MD5 sum of the received file.
I have used netcat a lot in this way, never seen it fail, never seen it fail to max out the network either..
I don't think finding something to transfer faster is your problem, double check to make sure your target file system is NOT FAT, as others have said. Also, make sure the NICs on both sides have updated drivers and aren't otherwise acting flaky.
With that said, are you moving many many small files or just a few large ones? I've seen RAID controller problems when trying to move millions of tiny files.
I don't think you'll have a problem automating this once you figure out what is causing the failure. It may help to list more details about your hardware and any relevant errors you may see in event viewer.
Have you tried to use an eSATA connection to an external hard drive? Connections are smokin' fast (3 gigabit!) and should be able to transfer that file in no time!
What speed is your network card on the server, 10/100 or 10/100/1000? What does the server network bandwidth and switch look like when you are coping the file? What does the destination location's (server's?) network bandwidth look like when copying? Have you tried to teamed 2 NIC's together? Are the network card drivers up to date? Is the BIOS up to date?
There are many things that could be the issue for file transfers. Making sure the hardware drivers and BIOS are up to date can really make a difference.