I've a Solaris 8 box which is reporting that a file system is full:
db% tail -2 /var/adm/messages Nov 22 08:32:27 db ufs: [ID 845546 kern.notice] NOTICE: alloc: /u03: file system full Nov 22 08:34:51 db last message repeated 12 times
But df says it's not out of free blocks, and it's not out of free inodes either:
db% df -k /u03 Filesystem kbytes used avail capacity Mounted on /dev/md/dsk/d6 282330903 254957403 24550191 92% /u03 db% df -o i /u03 Filesystem iused ifree %iused Mounted on /dev/md/dsk/d6 29663278 4230866 88% /u03
So I thought perhaps some process is holding open file descriptor(s) to 20+GB of deleted files. But lsof, sorting by size, reports nothing over 2GB, and that's a legitimate file:
db% lsof /u03 | sort -n +6 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME [...] oracle 1257 oracle 278u VREG 85,6 2097160192 9685782 /u03/oradata/(redacted)/data/foo_tab_14.dbf db%
I'd appreciate any pointers to any other resource save free blocks and inodes whose exhaustion can fill up a file system, or some other way to "hide" used blocks/inodes, or any other idea anyone has. Rebooting this box or shutting down oracle is not a valid investigative option.
Edit: Khaled, I couldn't at the time. I can't post the output, because one of the DBAs has freed up about 4GB which means the machine can continue functioning, and if I "fill it up" again as a test, things will break. But this is the second time in 24 hours it's got to about 92% full and "filled up" (as in, new files can't be created, and
/var/adm/messages reports "file system full"), and yes, when it happens it definitely breaks the creation or extension of files on that FS.