18

This question was previously asked in stack overflow but the good folks there have recommended that i try the community over here instead.

I am researching on sparse files with regards to various filesystems and am trying to find something concrete that states that sparse files is supported by Network File Systems (NFS) or Server Message Block (SMB).

I understand that SMB is widely used in Windows and that according to this entry, an SMB server can support sparse file even if the underlying file system does not. However, if i am right, the file system that does not support sparse files would just fill the 'holes' with zeroes and this could lead to a performance problem.

With regards to NFS, i have not been able to find out anything about using NFS supporting sparse files.

Hence, my questions is,

Are sparse files supported in NFS and SMB ?

12

NFS: it has a partial support for sparse file. Basically, it supports creating a sparse file but, when reading, the file is expanded to include zeroes. This means that, while you can create a sparse file via NFS, when reading back that very same file the in-transit network data will include any zeroes found on the original file. A simple test show that behavior:

cd /mnt/nfs
truncate test.img -s 1G
ls -lh test.img

-rw-r--r--. 1 root root 1.0G Oct 26 11:29 test.img

du -hs test.img

0 test.img

As you can see, the test.img file has an on-disk size of 0 bytes. However, reading back it using dd if=test.img of=/dev/null bs=1M iflag=direct shows

1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 10.2269 s, 105 MB/s

It is clear that when transferring the sparse file, it is expanded to include all zeros.

NFSv4.2 will expand by including special handling for network transfer of sparse file. In other word, with NFSv4.2 the above dd will complete almost instantly.

SMB: it has the same behavior as NFS, at least in my test environments, using a Samba v3.6.x server with CIFS v1 and a Linux client using mount.cifs. Maybe under Windows it behave differently...

  • Can NFS support sparse files if the NFS server's underlying file system does not support sparse file? – Andrew Henle Oct 26 '15 at 10:59
  • 2
    @shodanshok: your test is invalid. Executing the same commands on a file system that does support sparse files yields the same result. dd reads in block by block and whether the underlying file system supports sparse files or not, holes are turned into zeros by the OS. Try it on ext4 and you will see the same numbers. – abligh Oct 26 '15 at 13:01
  • @AndrewHenle if the underlying FS does not support sparse file, how can NFS expose a non-existant support? Anyway, nowadays it is quite difficult to find a filesystem without sparse file support, as all recent (ext3/4, xfs, etc) Linux filesystems support that feature. – shodanshok Oct 26 '15 at 15:41
  • 1
    @abligh You are wrong. Executing the dd command over a local sparse file will give much faster results. See here for an example:, root@hubble:~# truncate -s 1G test.img root@hubble:~# dd if=test.img of=/dev/null bs=1M iflag=direct 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 0.10478 s, 10.2 GB/s As you can see, reading a local sparse file give I/O speed north of 10 GB/s – shodanshok Oct 26 '15 at 15:46
  • 2
    @shodanshok - Oh I see, you are looking at the speed, not the amount transferred. Perhaps clarifying that in your answer would be helpful. The canonical test for a file being stored in a sparse way is du -s vs ls -l, but you are correct that doesn't help with the transmission over the network; but in either case (as strace will confirm) dd is reading the whole file, including 'holes' as zeroes, the difference only being where the 'zeroes' originate (server or client side). However note (as per my answer) that NFS 4.2 does fully support sparse files. – abligh Oct 26 '15 at 15:53
10

NFS

Yes, NFS 4.2 fully supports sparse files (see this canonical document and this presentation).

Prior to NFS 4.2, the NFS client/server model supported sparse files in the sense that the API supported all POSIX file operations. This meant that writing sparse files on a server which supported sparse files on the backing file system resulted in a sparse file being created (rather than storing lots of zeros). But reading the file would result in the transmission of a lot of zeroes for the sparse element. IE the answer is 'partially'.

NFS 4.2 adds the ability for the client to 'see' holes in the files, and therefore for the server to not have to transmit all those zeroes. From the ID:

1.4.3.  Sparse Files

Sparse files are ones which have unallocated or uninitialized data
blocks as holes in the file.  Such holes are typically transferred as
0s during I/O. READ_PLUS (see Section 15.10) allows a server to send
back to the client metadata describing the hole and DEALLOCATE (see
Section 15.4) allows the client to punch holes into a file.  In
addition, SEEK (see Section 15.11) is provided to scan for the next
hole or data from a given location.

Despite the fact the specification supports sparse files, it would be possible for a lazy implementor to avoid implementing support for sparse files in either the client or the server.

SMB

I know less about SMB, but I believe it does support sparse files too, if the relevant FS capability bit is set. See here for more info.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.