I have solved the mystery myself, with help of the aufs maintainer, Junjiro Okajima.
The first step, to debug the problem, was to reproduce it in a controlled way. It took me some time (now I wonder why so much) to find out, that the problem occurs when files are written and deleted via aufs.
Reproducing the problem
create mount points:
# cd /tmp
# mkdir rw
# mkdir mnt
mount the tmpfs:
# mount -t tmpfs none /tmp/rw
mount the aufs, overlaying /usr with /tmp/rw:
# mount -t aufs -n -o "br:/tmp/rw:/usr" none "/tmp/mnt"
now I can see /usr contents under /tmp/mnt:
# ls /tmp/mnt
bin games include lib lib64 local sbin share src
what I am interested in is the used/available space on the tmpfs below:
# du -sk /tmp/rw
0 /tmp/rw
# df /tmp/rw
Filesystem 1K-blocks Used Available Use% Mounted on
none 1031128 24 1031104 1% /tmp/rw
No files in /tmp/rw, but 24 blocks allocated. Still not a big problem.
I can write a file to the aufs, it will be stored on tmpfs in /tmp/rw:
# dd if=/dev/zero of=/tmp/mnt/test bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.000343903 s, 298 MB/s
# du -sk /tmp/rw
100 /tmp/rw
# df /tmp/rw
Filesystem 1K-blocks Used Available Use% Mounted on
none 1031128 128 1031000 1% /tmp/rw
Note how the usage stats changed. du
show 100kB added, as expected, but the 'Used' value in the df
output increased by 104 blocks.
When I remove the file:
# du -sk /tmp/rw
0 /tmp/rw
# df /tmp/rw
Filesystem 1K-blocks Used Available Use% Mounted on
none 1031128 28 1031100 1% /tmp/rw
Four blocks are lost.
When I repeat the dd
and rm
commands a few times I get:
# df /tmp/rw
Filesystem 1K-blocks Used Available Use% Mounted on
none 1031128 36 1031092 1% /tmp/rw
More and more tmpfs blocks were gone and I didn't know where…
Where I did the same – dd
and rm
directly on /tmp/rw nothing was lost this way. And after un-mounting the aufs, the lost space on tmpfs was recovered. So, at least, I knew it was aufs, not tmpfs to blame.
What has been happening
Knowing what to blame, I described my problem on the aufs-users mailing list. I have quickly received first answers. The one from J. R. Okajima helped me to explain what is happening to the missing tmpfs blocks.
It was a deleted file, indeed. It wasn't shown by lsof
or anywhere in /proc/<pid>/*
as the file was not opened or mmaped by any user-space process. The file, the 'xino file', is aufs' external inode number translation table and is used internally by the kernel aufs module.
Path to the file can be read from sysfs:
# cat /sys/fs/aufs/si_*/xi_path
/tmp/rw/.aufs.xino
But, as the file is deleted, it cannot be seen directly:
# ls -l /tmp/rw/.aufs.xino
ls: cannot access /tmp/rw/.aufs.xino: No such file or directory
Though, information about its size and sizes of the other special aufs files can be read from the debugfs:
# for f in /sys/kernel/debug/aufs/si_8c8d888a/* ; do echo -n "$f: " ; cat $f ; done
/sys/kernel/debug/aufs/si_8c8d888a/xi0: 1, 32x4096 132416
/sys/kernel/debug/aufs/si_8c8d888a/xi1: 1, 24x4096 626868
/sys/kernel/debug/aufs/si_8c8d888a/xib: 8x4096 4096
/sys/kernel/debug/aufs/si_8c8d888a/xigen: 8x4096 88
The details are described in the aufs manual page.
The solution
The 'xino file' can be manually truncated by:
# mount -o remount,itrunc_xino=0 /tmp/mnt
Automatic xino file truncation can be requested by using trunc_xino option while mounting the aufs:
# mount -t aufs -n -o "br:/tmp/rw:/usr,trunc_xino" none "/tmp/mnt"
I still don't know how does it affect file system performance or if this will really solve my out-of-tmpfs-space problems on production… but I have learned a lot.
df
command. You can edit your answer to provide requested information.