Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've a Solaris 8 box which is reporting that a file system is full:

db% tail -2 /var/adm/messages
Nov 22 08:32:27 db ufs: [ID 845546 kern.notice] NOTICE: alloc: /u03: file system full
Nov 22 08:34:51 db last message repeated 12 times

But df says it's not out of free blocks, and it's not out of free inodes either:

db% df -k /u03
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d6       282330903 254957403 24550191    92%    /u03
db% df -o i /u03
Filesystem             iused   ifree  %iused  Mounted on
/dev/md/dsk/d6      29663278 4230866    88%   /u03

So I thought perhaps some process is holding open file descriptor(s) to 20+GB of deleted files. But lsof, sorting by size, reports nothing over 2GB, and that's a legitimate file:

db% lsof /u03 | sort -n +6
COMMAND     PID     USER   FD   TYPE DEVICE   SIZE/OFF     NODE NAME
[...]
oracle     1257   oracle  278u  VREG   85,6 2097160192  9685782 /u03/oradata/(redacted)/data/foo_tab_14.dbf
db%

I'd appreciate any pointers to any other resource save free blocks and inodes whose exhaustion can fill up a file system, or some other way to "hide" used blocks/inodes, or any other idea anyone has. Rebooting this box or shutting down oracle is not a valid investigative option.

Edit: Khaled, I couldn't at the time. I can't post the output, because one of the DBAs has freed up about 4GB which means the machine can continue functioning, and if I "fill it up" again as a test, things will break. But this is the second time in 24 hours it's got to about 92% full and "filled up" (as in, new files can't be created, and /var/adm/messages reports "file system full"), and yes, when it happens it definitely breaks the creation or extension of files on that FS.

share|improve this question
    
Can you create files under the full partitions or is it just an annoying message? –  Khaled Nov 22 '11 at 10:25
add comment

1 Answer

up vote 2 down vote accepted

Check the nbfree value with

fstyp –v | head –18

If that says some low value, this blog post I found might help you. I quote the beginning of the post:

At work we have a Solaris 8 with a UFS which told the appli­ca­tion that it can not cre­ate new files. The df com­mand showed plenty if free inodes, and there was also enough space free in the FS. The rea­son that the appli­ca­tion got the error was that while there where still plenty of frag­ments free, no free block was avail­able any­more. You can not cre­ate a new file only with frag­ments, you need to have at least one free block for each new file.

To see the num­ber of free blocks of a UFS you can call “fstyp –v | head –18″ and look at the value behind “nbfree”.

share|improve this answer
    
By the way, I've had almost identical situation for JFS3 filesystem once. –  kubanczyk Nov 22 '11 at 10:39
    
Janne, you're a star, I suspect this is it. Next time we go "FS full" I've got the command sitting at a prompt, ready to run. News as it's made. –  MadHatter Nov 22 '11 at 10:56
    
Great! Let me know how this goes :) –  Janne Pikkarainen Nov 22 '11 at 11:07
    
Ugh, what a time for my DBAs to get a conscience, and start clearing up the FS wholesale. From about 900,000 free blocks when I first looked, it's now up to just north of three million. Although that means I can't confirm your hypothesis, the nature of the increase makes me fairly sure you're right, so I'm going to accept your answer now; if at some point the DBAs let the error happen again, and nbfree is not zero or very close to it, I'll come back and reopen the issue. I hope that's OK. –  MadHatter Nov 24 '11 at 10:56
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.