I have to copy (periodically) a MSSQL backup file large 500 GB to another pc (windows or linux) within lan.

The normal copy (or xcopy, robocoy, teracopy) command to a network share (or external usb disk) takes a lot of time (about 12 hours) and the server with the backup file become very slow (because there is only one RAID SAS controller).

An "idea" maybe to split the backup file into smaller parts (like torrent) and copy these parts concurrently, but I don't know how to do it.

My primary target is to copy without impact the source server.

Any idea or tool suggested will be appreciated.

  • There's a good chance that copying smaller parts at the same time will reduce performance as the machine has to start moving the heads more often to chase down multiple sections at the same time (not to mention the CPU and network overhead from multitasking). – Brian Knoblauch Sep 3 '10 at 15:05

With windows, unfortunately, your options are a bit more limited. You could simply try setting up a FTP server (don't use the IIS one) and set a max bandwidth. This should effectively lower the impact on the server.

In all honesty however, If you're running into a noticeable slow-down when doing a single-threaded copy of a single file... I suspect that there are other issues at play. If you're talking about copying from SAS -> USB (or vise-versa) and the lag is happening... There is probably some issue with the USB stack. The USB stack shouldn't affect system performance that drastically. If you are running into the lag when doing ANYTHING from the SAS volume... perhaps the firmware of your raid controller is b0rked... or you're using the wrong drivers... (or in some cases... a REALLY cheap SAS controller). It can also be the result of some anti-virus program (or multiple programs overlapping) causing the machine to work 4x as hard as necessary. (as a rule... only use 1 anti-virus program. More will just make your machine work harder for no reason.) Even having your volume fragmented all over the place can cause the symptoms you're talking about.

I would first try to fix the problem rather than simply work around it.

  • The SAS controller works well, the problem is that the server has 6 disks SAS (RAID1 and RAID0) managed by only one controller; and the RAID1 volumes are used as data partition for a database server with high traffic; and the large file to copy is on RAID0 partition. – lg. Sep 3 '10 at 14:16
  • I will test the ftp solution. – lg. Sep 3 '10 at 14:16
  • Seconded. Also, if the problems are related to IO on your disk/controller, splitting the file on that volume will cause the same performance problems, and then copying multiple smaller files from that volume will probably cause even worse performance problems. – mfinni Sep 3 '10 at 14:17
  • Crap. Seriously. I have ONE raid controller running 13 discs in a computer and I can totally copy nearly a gigabyte of data per second without the LAN getting slow (on the discs). 1gb network is only 100mb/s and should not overload a high end RAID 1 or RAID 0. – TomTom Sep 3 '10 at 15:02
  • SOMETHING is wrong here. – TomTom Sep 3 '10 at 15:02

By your description, the server cannot cope with the I/O load. I would be inclined to think that's due to some other fault as TheCompWiz said. Until/unless you fix/alleviate the cause of the problem (you think it's insufficient spindles or too slow a RAID controller) you will not get it copied across any quicker.

As your foremost intent is to copy it without impacting the server, look at throttling bandwidth/copy speed, i.e. deliberately making it run slower. robocopy has an "inter-packet gap" option robocopy /ipg:N (where N is some number) which can be used to limit speed. http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx. I understand teracopy does too.

To clarify do you need to copy this file only once, or are you needing to copy a backup file periodically on a schedule? If it's periodically, for backup purposes, then as Tom said, copying only what has changed from one 500GB file to the next 'version' will save a great deal of time and put less load on the server. Perhaps if you explained your overall intent rather than the bit you're currently stuck you might get some useful ideas.

  • I will try the robocopy option. – lg. Sep 3 '10 at 14:59
  • I need to copy a MSSQL backup file periodically. – lg. Sep 3 '10 at 15:01
  • +1 for robocopy option – Nick Kavadias Sep 3 '10 at 15:06

the only way to really make this go alot faster is to go to some kind of block level solution that would only ever copy the deltas.

  • Can you suggest me a tool? Better if open source or freeware. – lg. Sep 3 '10 at 15:04

Why put lots of individual files into one big one then try to split it up again?

Backup the individual files.

  • This is not my case. The backup file is generated by MSSQL server. – lg. Sep 3 '10 at 14:57

Have you monitored memory usage on the windows server while doing the copy? I wonder if you're running into the "large files" issue.

Have a look at Windows Server 2008 x64, Large File Transfers, and Memory Usage (and some of the other pages linked from there)


Other solutions: Use differential or transactional backup instead of full backup. - This will cut down on the amount of data you will need to transfer.

CDP: I use R1soft http://www.r1soft.com/ This software has saved my ass so many times.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.