I have tested a 10Gbit connection between two hosts to be able to read a 10GB file from host1 and write it to host2 using netcat, where the speed was 410MB/s.
When I ZFS send/receive again with netcat over the same dedicated 10Gbit connection, I only get 70MB/s. The snapshot was 2.5TB with 15 million files.
What could the reason for this slowdown be? Is the bottleneck that it takes a lot of time to rollback a snapshot with this many files, or does number of files not have an influence of ZFS rollback speed?
The 10GB file transfer test where I got 410MB/s do I suppose simulates a ZFS send/receive with rollback. So with that assumption I am very surprised that I see so different speeds. I am using speed for comparison between the two tests, so I don't have to generate a 2.5TB file with random data.
So I don't understand why "read file from host1, transfer with netcat, write file to host2" is much faster than "zfs send snapshot from host1, transfer with netcat, ZFS receive/rollback on host2".
Maybe another way to ask the same would be?:
If I have two 2,5TB snapshots of equal size, where snapshot1 contains 1 file, and snapshot2 contains 15 million files. Will the time for
zfs receive for both of them be the same? Or will one be faster than the other?