Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Hey everyone, first question around here.

I have a centos 5.2 server and running df -h i get this:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      672G -551M  638G   0% /
/dev/hda1              99M   12M   82M  13% /boot
tmpfs                 2.0G     0  2.0G   0% /dev/shm

that space wasn't even near 10% usage the last time it showed a correct value, i'm at a loss with whats going on.

Thanks.

Edit:

Ok so i had to reboot the server cause ssh hanged up, i'm guessing it was related to this.

Some new info, after reboot df -h showed 12Gb used (2%), but if i run du -hcs / it shows 46Gb total, theres a big difference here

Edit2:

After about 15mins of uptime df -h is showing negative values again:

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 672G -24G 660G - /

Edit3:

More info, ran a fsck and this is the output:

Checking all file systems.
  [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -f -y /dev/VolGroup00/LogVol00 
Pass 1: Checking inodes, blocks, and sizes 
Pass 2: Checking directory structure 
Pass 3: Checking directory connectivity 
Pass 4: Checking reference counts 
Pass 5: Checking group summary information
 /dev/VolGroup00/LogVol00: 204158/181633024 files (1.3% non-contiguous), 9224806/181633024 blocks 
[/sbin/fsck.ext3 (1) -- /boot] fsck.ext3 -f -y /dev/hda1 
Pass 1: Checking inodes, blocks, and sizes 
Pass 2: Checking directory structure 
Pass 3: Checking directory connectivity 
Pass 4: Checking reference counts 
Pass 5: Checking group summary information 
/boot: 34/26104 files (5.9% non-contiguous), 15339/104388 blocks
share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

I think it is a file system corruption. You should unmount the partition and run a fsck.
Check also the logs and the console for any file system errors.

share|improve this answer
    
so this logical group is mounted on /, is it possible to safely run a fsck without having physical access? –  GriffinHeart Jun 14 '10 at 16:28
    
you have to stop all processes having files opened for writing and then remount / read-only. So you have to stop pretty everything. I would experiment this on a VM with the same OS installed. To mount the root file system read-only: mount / -o remount,ro. After you run the fsck with file system mounted read-only you have to remount it read-write and start the daemons, or better just reboot/reset. –  Mircea Vutcovici Jun 14 '10 at 20:11
    
i've ran fsck and from the output it doesn't seem anything is wrong, i'll update with the log –  GriffinHeart Jun 15 '10 at 22:19
    
May be you have a hardware problem. Run a memtest86+. For this you will have to reboot the server. And the downtime will be a few good hours. Run other hardware tests... for CPU, etc... Check the temperatures on the server. –  Mircea Vutcovici Jul 2 '10 at 22:10
add comment

I think this might mean that you have gone beyond what is reserved as a root only space (Default is 5% on ext3 I think):

$ sudo tune2fs -l /dev/sda1 | grep -i 'Reserved block count'
Reserved block count:     1877194

Reserved block count is a certain amount of blocks that only the root user can use after the disk is almost full (This prevents a normal user from filling up the fs and causing things to break). From man tune2fs:

   -m reserved-blocks-percentage 

Set the percentage of the filesystem which may only be allocated by privileged processes. Reserving some number of filesystem blocks for use by privileged processes is done to avoid filesystem fragmentation, and to allow system daemons, such as syslogd(8), to continue to function correctly after non-privileged processes are prevented from writing to the filesystem. Normally, the default percentage of reserved blocks is 5%.

So I think something is taking up space fast as the root user. You can use du -hcs / and drill down from there to find where the files are that are using the space. If you think it might be something creating large files, you could also use the find command.

share|improve this answer
    
Thanks, i've added additional info. I'm still trying to troubleshoot what's happening. –  GriffinHeart Jun 14 '10 at 15:46
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.