File system full - except it's not

Question

I've a Solaris 8 box which is reporting that a file system is full:

db% tail -2 /var/adm/messages
Nov 22 08:32:27 db ufs: [ID 845546 kern.notice] NOTICE: alloc: /u03: file system full
Nov 22 08:34:51 db last message repeated 12 times

But df says it's not out of free blocks, and it's not out of free inodes either:

db% df -k /u03
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d6       282330903 254957403 24550191    92%    /u03
db% df -o i /u03
Filesystem             iused   ifree  %iused  Mounted on
/dev/md/dsk/d6      29663278 4230866    88%   /u03

So I thought perhaps some process is holding open file descriptor(s) to 20+GB of deleted files. But lsof, sorting by size, reports nothing over 2GB, and that's a legitimate file:

db% lsof /u03 | sort -n +6
COMMAND     PID     USER   FD   TYPE DEVICE   SIZE/OFF     NODE NAME
[...]
oracle     1257   oracle  278u  VREG   85,6 2097160192  9685782 /u03/oradata/(redacted)/data/foo_tab_14.dbf
db%

I'd appreciate any pointers to any other resource save free blocks and inodes whose exhaustion can fill up a file system, or some other way to "hide" used blocks/inodes, or any other idea anyone has. Rebooting this box or shutting down oracle is not a valid investigative option.

Edit: Khaled, I couldn't at the time. I can't post the output, because one of the DBAs has freed up about 4GB which means the machine can continue functioning, and if I "fill it up" again as a test, things will break. But this is the second time in 24 hours it's got to about 92% full and "filled up" (as in, new files can't be created, and /var/adm/messages reports "file system full"), and yes, when it happens it definitely breaks the creation or extension of files on that FS.

Answer 1

Check the nbfree value with

fstyp –v | head –18

If that says some low value, this blog post I found might help you. I quote the beginning of the post:

At work we have a Solaris 8 with a UFS which told the appli­ca­tion that it can not cre­ate new files. The df com­mand showed plenty if free inodes, and there was also enough space free in the FS. The rea­son that the appli­ca­tion got the error was that while there where still plenty of frag­ments free, no free block was avail­able any­more. You can not cre­ate a new file only with frag­ments, you need to have at least one free block for each new file.

To see the num­ber of free blocks of a UFS you can call “fstyp –v | head –18″ and look at the value behind “nbfree”.