I have some test code that generates a list of 5000 small blocks of data (min. size: 351 bytes, max size: 386 bytes, average size: 385 bytes) and then writes them as separate files to a directory - average write time is around 5 msec per file
If I use the same exact code but instead of writing the 5000 different blocks I repeat the same data over and over (e.g. writing for 500 times the first 10 block) the average write time goes does to less than 2 msec per file - as if in some way the file system is able to optimize the writes because the data is the same.
Does anyone have an explanation for this behavior?
Testing on a Surface Pro 4 - i5 processor with 8 GB of RAM, writing to the built-in solid state drive. Operating system is Windows 10.
Test code:
const int count = 5_000;
// Generate a list of count small byte arrays: min. size: 351, max size: 386, average size: 385
var bytes = SerializeObjects( count );
// Write them all to disk as individual files
var watch = Stopwatch.StartNew();
for ( var i = 0; i < count; i++ )
{
File.WriteAllBytes(
Path.Combine( _directory, Guid.NewGuid() + ".xml" ),
bytes[ i ]
);
}
watch.Stop();
// Timed at: around 5ms per file
Console.WriteLine( "Wrote {0:n0} files in {1:n0} ms ({2:n3} ms per file)", count, watch.ElapsedMilliseconds, (double)watch.ElapsedMilliseconds / count );
modifying the WriteAllBytes()
call to write bytes[ i % 10 ]
(so repeating the first 10 values over and over again) the time per file goes down to less than 2 msec
Update
It is not dedup:
PS > Get-DedupProperties C
Get-DedupProperties : Deduplication feature is not available