We have a Windows Server 2003 based server sharing a folder through a gigabit interface that receives a 10gb file every hour from another server. This server is a Dell PowerEdge 830 with two SCSI disks on RAID-1 with a PERC4/SC adapter. The transfer occurs on almost 10 minutes, and the server becomes unusable during the transfer period. We can't even open Windows Explorer, Task Manager or access the server via VNC. We need some advice about what the problem is, or how to track further to find the problem.
2 Answers
This is almost certainly due to the write buffer on the server inflating because the inbound transfer exceeds the write rate of the disk subsystem - see Mark Russinovitch's article on network copy behavior in various Windows Versions here.
One of the biggest problems with the engine’s implementation is that for copies involving lots of data, the Cache Manager write-behind thread on the target system often can’t keep up with the rate at which data is written and cached in memory. That causes the data to fill up memory, possibly forcing other useful code and data out, and eventually, the target’s system’s memory to become a tunnel through which all the copied data flows at a rate limited by the disk.
The engine he is talking about here is the one from Windows XP (and W2K3).
Once the buffering exceeds the total amount of free physical RAM on the system you're going to see physical paging in addition to the writes associated with the copy, and you also have all standard Windows IO activity still trying to proceed. As soon as you try to initiate a new process that triggers further paging - slowing everything down again.
You may have an issue with your disks but even under ideal conditions it's unlikely that a RAID 1 config on this server could sustain a 10Gig write stream much faster than about 40-50Meg/sec while your GigE link will almost certainly exceed that (and could easily be double that, if not slightly more).
Your options are:
- Add more disks so that the array is faster - you'll need to go to at least a 4 disk RAID 10 or 6/7 disk RAID 5 to get your streaming write IO high enough to avoid buffering.
- Increase RAM enough to buffer the whole stream, or at least a large chunk of it if your version of Windows Server and your hardware allows that.
- Find some way to throttle the transfer rate down to around 10-20Meg/sec.
- Upgrade to W2K8 (which has a smarter network copy buffer).
This sounds very familiar -- I've been wrestling with a similar issue. See these two threads:
Copying large file to remote server causes it to run out of physical memory
Windows Server 2008 x64, Large File Transfers, and Memory Usage
