df shows negative values for used

Question

Hey everyone, first question around here.

I have a centos 5.2 server and running df -h i get this:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      672G -551M  638G   0% /
/dev/hda1              99M   12M   82M  13% /boot
tmpfs                 2.0G     0  2.0G   0% /dev/shm

that space wasn't even near 10% usage the last time it showed a correct value, i'm at a loss with whats going on.

Thanks.

Edit:

Ok so i had to reboot the server cause ssh hanged up, i'm guessing it was related to this.

Some new info, after reboot df -h showed 12Gb used (2%), but if i run du -hcs / it shows 46Gb total, theres a big difference here

Edit2:

After about 15mins of uptime df -h is showing negative values again:

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 672G -24G 660G - /

Edit3:

More info, ran a fsck and this is the output:

Checking all file systems.
  [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -f -y /dev/VolGroup00/LogVol00 
Pass 1: Checking inodes, blocks, and sizes 
Pass 2: Checking directory structure 
Pass 3: Checking directory connectivity 
Pass 4: Checking reference counts 
Pass 5: Checking group summary information
 /dev/VolGroup00/LogVol00: 204158/181633024 files (1.3% non-contiguous), 9224806/181633024 blocks 
[/sbin/fsck.ext3 (1) -- /boot] fsck.ext3 -f -y /dev/hda1 
Pass 1: Checking inodes, blocks, and sizes 
Pass 2: Checking directory structure 
Pass 3: Checking directory connectivity 
Pass 4: Checking reference counts 
Pass 5: Checking group summary information 
/boot: 34/26104 files (5.9% non-contiguous), 15339/104388 blocks

Answer 1

I think it is a file system corruption. You should unmount the partition and run a fsck.
Check also the logs and the console for any file system errors.

Answer 2

I think this might mean that you have gone beyond what is reserved as a root only space (Default is 5% on ext3 I think):

$ sudo tune2fs -l /dev/sda1 | grep -i 'Reserved block count'
Reserved block count:     1877194

Reserved block count is a certain amount of blocks that only the root user can use after the disk is almost full (This prevents a normal user from filling up the fs and causing things to break). From man tune2fs:

   -m reserved-blocks-percentage 

Set the percentage of the filesystem which may only be allocated by privileged processes. Reserving some number of filesystem blocks for use by privileged processes is done to avoid filesystem fragmentation, and to allow system daemons, such as syslogd(8), to continue to function correctly after non-privileged processes are prevented from writing to the filesystem. Normally, the default percentage of reserved blocks is 5%.

So I think something is taking up space fast as the root user. You can use du -hcs / and drill down from there to find where the files are that are using the space. If you think it might be something creating large files, you could also use the find command.