We are running a load balanced cluster of three web servers, all running Windows 2008. On these machines we are hosting a bunch of e-commerce sites, all with several thousands articles. This adds up to about 100k+ media files each, counting all sizes etc. Probably 1.5M+ files in total, of different sizes, ranging from 1kb to 10mb. All "master" files are on a file server (also windows 2008).
In the beginning of time, we used a UNC path for including media for the webserver. This was however not a very good idea as traffic started to pickup.
We then switched to a periodically running script that utilized robocopy, but this gave very heavy load on the disks on the web servers, actually slowing web requests down quite a bit.
We then evolved the script, so that on the file server, it sorts out the newest files (say, a day old or so) and copy them to a staging folder, and then copy the files from that folder out to the web servers. The load on the web servers got down to an acceptable level with this, but the file server is choked up.
We then decided to try DFS-R out, even though I previously had very bad experiences with it. We moved some of the customers folders to DFS-R replicated ones, and it worked for a few days, when suddenly one of our biggest customers media folders suddenly emptied, for no apparent reason. This resulted in a total loss of media and we had to restore from backup. This must be the number one priority in debugging, THIS SHOULD NOT HAPPEN. But it happened.
My question to you is: What is the next step? As you can see, this is a fairly big setup, with 4 servers in replication with eachother. The replication does only need to be one-way, from the file server out to the web servers. More servers will be added.
