0

We are using ext4 filesystems to store data in only one big file.

We are having a strange issue : df reports free space (16MB), dumpe2fs reports 4096 free blocks (of 4KB, matches what df reports), but when trying to append data to our file the system reports No space left on device.

[root@localhost raw_2]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdd1        11G   11G   16M 100% /opt/HIDDEN/storage/raw_2
[root@localhost raw_2]# df -i .
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sdd1       11264    12 11252    1% /opt/HIDDEN/storage/raw_2
[root@localhost raw_2]# echo test > testfile
-bash: echo: write error: No space left on device
[root@localhost storage]# umount /dev/sdd1
[root@localhost storage]# dumpe2fs -h /dev/sdd1
dumpe2fs 1.41.12 (17-May-2010)
Filesystem volume name:   <none>
Last mounted on:          /opt/HIDDEN/storage/raw_2
Filesystem UUID:          9d6ca417-0854-461d-993d-e23c2a2229d4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              11264
Block count:              2883580
Reserved block count:     0
Free blocks:              4096
Free inodes:              11251
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      703
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         128
Inode blocks per group:   8
Flex block group size:    2048
Filesystem created:       Tue Nov 15 11:32:52 2016
Last mount time:          Tue Dec  6 14:14:57 2016
Last write time:          Tue Dec  6 15:16:54 2016
Mount count:              8
Maximum mount count:      32
Last checked:             Tue Nov 15 11:32:52 2016
Check interval:           15552000 (6 months)
Next check after:         Sun May 14 12:32:52 2017
Lifetime writes:          3572 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      dc8da308-ef45-4bea-b156-5690eb399646
Journal backup:           inode blocks
Journal features:         (none)
Journal size:             128M
Journal length:           32768
Journal sequence:         0x0004a45b
Journal start:            0

This is not due to reserved blocks for root and obviously not related to free inodes.

I have read a lot of discussions but found nothing that could apply to our case.

This problem happens on a VM with 11GB virtual disks. We have also seen this same behavior when using 10GB files mounted as loopback FS. We are using RedHat 6 (linux 2.6.32).

Our software is running as root. I also tried a simple cat /dev/zero > testfile, it stopped at the same limit, leaving 16MB free space.

Any help would be appreciated.

EDIT : using statfs on 2 different systems shows a strange behavior.

On my original system, statfs does not report any difference f_bfree==f_bavail :

statfs("/opt/HIDDEN/storage/raw_1/", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=2882680, f_bfree=2665715, f_bavail=2665715, f_files=11264, f_ffree=11251, f_fsid={-2032429898, 1911081970}, f_namelen=255, f_frsize=4096}) = 0

On a debian testing box (linux 4.8.0) :

statfs("/home/", {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=55137925, f_bfree=26426280, f_bavail=26422184, f_files=10114944, f_ffree=9685471, f_fsid={1010187930, 3946494061}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0

This fs has no reserved blocks, but there is a difference of 4096 blocks between f_bavail (free blocks available to unprivileged user) and f_bfree (free blocks in fs). Does anybody know what are these 4096 reserved blocks ?

Thanks.

1

Nearly all filesystems, ext4 included, reserve some space for their metadata, journal and fragment handling. You are likely seeing the effect of that reserved/unusable space.

Anyway, running a filesystem with true 100% utilization is a very bad idea. Always reserve some free space for an healthy filesystem.

UPDATE: this is a quote from the kernel mailing list explaining why exactly 4096 blocks are reseved for filesystem use:

And this is where reserved space comes into the picture. When mounting the file system we slice off a little bit of the file system space (2% or 4096 clusters, whichever is smaller) which can be then used for the cases mentioned above to prevent costly zeroout, or unexpected ENOSPC.

The number of reserved clusters can be set via sysfs, however it can never be bigger than number of free clusters in the file system.

  • Journal is 128MB on this FS (see at the end of dumpe2fs), this space is not accounted in free space. I suppose this is due to some metadata, but I would like to know exactly which metadata. I know this is a bad idea, but our software was designed this way, I have to do with it for the moment. – MCE35 Dec 6 '16 at 17:40
  • I've updated my answer with a clear explaination of what happens to your last 16 MB (4096 4K blocks) of space. – shodanshok Dec 6 '16 at 21:13
  • OK that's it, thank you ! The last thing is why kernel 2.6.32 is reporting these blocks is f_bavail and not kernel 4.8. I suppose this is a bug in older kernels. – MCE35 Dec 7 '16 at 8:52
  • Older kernels probably do not have the patch applied, so their behavior is different. – shodanshok Dec 7 '16 at 8:57
2

/dev/sdd1 11G 11G 16M 100% /opt/HIDDEN/storage/raw_2

100% full means 100% full. Nothing more can be written to this partition until you are less that 100%. I would venture a guess that the 16MB of free space is fragmented free space and not actually contiguous.

  • 100% is just rounded by df, it is actually 1-free_space/total_space which is 99.86% in my case, df shows it as 100%. Fragmentation should not occur as we are writing a single file continuously, but even if FS is fragmented does not mean space cannot be used (but performance may be degraded). – MCE35 Dec 6 '16 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.