NTFS - writing small files performance variations

Question

I have some test code that generates a list of 5000 small blocks of data (min. size: 351 bytes, max size: 386 bytes, average size: 385 bytes) and then writes them as separate files to a directory - average write time is around 5 msec per file

If I use the same exact code but instead of writing the 5000 different blocks I repeat the same data over and over (e.g. writing for 500 times the first 10 block) the average write time goes does to less than 2 msec per file - as if in some way the file system is able to optimize the writes because the data is the same.

Does anyone have an explanation for this behavior?

Testing on a Surface Pro 4 - i5 processor with 8 GB of RAM, writing to the built-in solid state drive. Operating system is Windows 10.

Test code:

const int count = 5_000;

// Generate a list of count small byte arrays: min. size: 351, max size: 386, average size: 385
var bytes = SerializeObjects( count );
// Write them all to disk as individual files
var watch = Stopwatch.StartNew();
for ( var i = 0; i < count; i++ )
{
    File.WriteAllBytes(
        Path.Combine( _directory, Guid.NewGuid() + ".xml" ),
        bytes[ i ]
    );
}
watch.Stop();
// Timed at: around 5ms per file
Console.WriteLine( "Wrote {0:n0} files in {1:n0} ms ({2:n3} ms per file)", count, watch.ElapsedMilliseconds, (double)watch.ElapsedMilliseconds / count );

modifying the WriteAllBytes() call to write bytes[ i % 10 ] (so repeating the first 10 values over and over again) the time per file goes down to less than 2 msec

Update

It is not dedup:

PS > Get-DedupProperties C
Get-DedupProperties : Deduplication feature is not available

Answer 1

Windows versions since Server 2012 support data "deduplication". It is possible this is enabled on your system and if so the OS is detecting the duplicate writes and is allocating "pointers" to the common file system data. The effect of this would be that NT would only be updating directory blocks and not having to allocate and write new file system blocks for the duplicate files thus significantly reducing I/O.