Data syncing behavior for filesystems in Windows

Question

In Linux mounting supports commit option, which is described in mount(8) man page as:

commit=nrsec
       Sync all data and metadata every nrsec seconds. The default
       value is 5 seconds. Zero means default.

In Windows write-back caching can work only if you have Removal policy set to Better performance in storage's Policies. Let's assume it is turned on.

Is there any way to similarly tune up write caching behavior in Windows as it can be done in Linux?

---

In Linux beside commit option, that must be supported by mounted filesystem type to take effect, there are also a few important VM settings that create additional constraints with regard to write-back cache behavior and can be changed by echoing new values to particular files in /proc/sys/vm/ directory, described in Documentation for /proc/sys/vm/* as:

• dirty_background_bytes

  Contains the amount of dirty memory at which the pdflush background writeback daemon will start writeback.

  Note: dirty_background_bytes is the counterpart of dirty_background_ratio. Only one of them may be specified at a time. When one sysctl is written it is immediately taken into account to evaluate the dirty memory limits and the other appears as 0 when read.

• dirty_background_ratio

  Contains, as a percentage of total system memory, the number of pages at which the pdflush background writeback daemon will start writing out dirty data.

• dirty_bytes

  Contains the amount of dirty memory at which a process generating disk writes will itself start writeback.

  Note: dirty_bytes is the counterpart of dirty_ratio. Only one of them may be specified at a time. When one sysctl is written it is immediately taken into account to evaluate the dirty memory limits and the other appears as 0 when read.

  Note: the minimum value allowed for dirty_bytes is two pages (in bytes); any value lower than this limit will be ignored and the old configuration will be retained.

• dirty_expire_centisecs

  This tunable is used to define when dirty data is old enough to be eligible for writeout by the pdflush daemons. It is expressed in 100'ths of a second. Data which has been dirty in-memory for longer than this interval will be written out next time a pdflush daemon wakes up.

• dirty_ratio

  Contains, as a percentage of total system memory, the number of pages at which a process which is generating disk writes will itself start writing out dirty data.

• dirty_writeback_centisecs

  The pdflush writeback daemons will periodically wake up and write `old' data out to disk. This tunable expresses the interval between those wakeups, in 100'ths of a second.

  Setting this to zero disables periodic writeback altogether.

(In fact pdflush threads are no longer available in Linux, as they were replaced by per backing device info flushing threads a few years ago, but it's subtlety. You can read more about it in LWN.net article Flushing out pdflush if you're interested.)

For the sake of completeness, current defaults in Linux kernel v3.2 are (checked on debian wheezy):

/proc/sys/vm/dirty_background_bytes:0
/proc/sys/vm/dirty_background_ratio:10
/proc/sys/vm/dirty_bytes:0
/proc/sys/vm/dirty_expire_centisecs:3000
/proc/sys/vm/dirty_ratio:20
/proc/sys/vm/dirty_writeback_centisecs:500

Does Windows expose any similar settings for its VM subsystem or maybe on per drive/partition basis?

Answer 1

I can tell you right off the bat that you won't be able to configure the standard Microsoft storage driver with that much granularity.

This is a difficult question, because the Memory Manager and the Cache Manager and the disk system in Windows are all constantly evolving. Advice that you get for Windows 2000 may be irrelevant for Windows 2003 which may be irrelevant for Windows 2008, etc. On top of that, you also have applications such as SQL Server with comprehensive cache managers of their own that may be totally unaffected by or react unpredictably to any changes you make to the OS file system cache. On top of that, this is the sort of functionality that would be taken over by a third party storage driver if you used one. You can assume that the Microsoft-provided one is more generic and therefore less configurable than a driver you might get from Intel or HP that is specialized for their hardware. Many vendor drivers require that Windows write-caching be disabled in order to enable their own write caching, etc.

All that said, I might be able to help you. Since you didn't specify what version of Windows, and like I said, things like registry paths and values and functions can and do totally change between versions of Windows, I'm just going to assume 2008R2/Win7.

From this MS article:

By default, Windows caches file data that is read from disks and written to disks. This implies that read operations read file data from an area in system memory known as the system file cache, rather than from the physical disk. Correspondingly, write operations write file data to the system file cache rather than to the disk, and this type of cache is referred to as a write-back cache. Caching is managed per file object.

Caching occurs under the direction of the cache manager, which operates continuously while Windows is running.

When I alter those checkboxes in the Windows Device Manager for "Enable write caching on the device" and "Turn off Windows write-cache buffer flushing," those changes are reflected in the registry values in HKLM\SYSTEM\CurrentControlSet\Enum\<IDE>\<DiskName>\<Serial Number>\Device Parameters\Disk\CacheIsPowerProtected = 1, etc. The values I placed in brackets are variables.

But I digress. You want to alter the behavior of the file cache. Such as making it flush more or less frequently.

You said you were interested in API calls, which is good, because I think you will be interested in Microsoft Windows Dynamic Cache Service. I think it was written by this guy. Sample source code is included.

A caveat for Win7/2008R2:

The memory management algorithms in Windows 7 and Windows Server 2008 R2 operating systems were updated to address many file caching problems found in previous versions of Windows. There are only certain unique situations when you need to implement this service on computers that are running Windows 7 or Windows Server 2008 R2.

Essentially, you'll be using the SetSystemFileCacheSize() Win32 API function. Check this out:

MinimumFileCacheSize [in]

The minimum size of the file cache, in bytes. The virtual memory manager attempts to keep at least this much memory resident in the system file cache.

To flush the cache, specify (SIZE_T) -1.

This implies you can flush the cache at will, on whatever schedule pleases you.

Good luck!