Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a runing centos 6, which suddenly fail due to hd full... which cause us sever problems since several days.

when i do:

df -h
S.ficheros            Size  Used Avail Use% Montado en
rootfs                9,7G  9,1G  158M  99% /
/dev/root             9,7G  9,1G  158M  99% /
none                  7,8G  404K  7,8G   1% /dev
/dev/md2              101G  188M   96G   1% /home
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/var/named
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.conf
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.rfc1912.zones
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/rndc.key
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/usr/lib64/bind
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.iscdlv.key
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.root.key

And when i go ahead, I get:

du -sh /var/named/*
du: ATENCIÓN: Estructura de directorios circular.
Esto quiere decir seguramente que el sistema de ficheros está corrupto.
COMUNÍQUELO AL ADMINISTRADOR DEL SISTEMA.
El siguiente directorio es parte del ciclo:
«/var/named/chroot/var/named/chroot»

268K /var/named/chroot
144K /var/named/data
12K /var/named/dynamic
4,0K /var/named/named.ca
4,0K /var/named/named.empty
4,0K /var/named/named.localhost
4,0K /var/named/named.loopback
4,0K /var/named/slave

So, i reboot into rescue mode, then i did:

fdisk -l

Disk /dev/sda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0000bd74

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1275    10238976+  fd  Linux raid autodetect
/dev/sda2            1275       14528   106451968   fd  Linux raid autodetect
/dev/sda3           14528       14593      523264   82  Linux swap / Solaris

Disk /dev/sdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0000fef6

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1275    10238976+  fd  Linux raid autodetect
/dev/sdb2            1275       14528   106451968   fd  Linux raid autodetect
/dev/sdb3           14528       14593      523264   82  Linux swap / Solaris

Disk /dev/md2: 109.0 GB, 109006749696 bytes
2 heads, 4 sectors/track, 26612976 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md2 doesn't contain a valid partition table

Disk /dev/md1: 10.5 GB, 10484645888 bytes
2 heads, 4 sectors/track, 2559728 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

So, i made the fsck as below:

root@rescue:~# fsck -yvf /dev/md2 
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

      12 inodes used (0.00%)
       0 non-contiguous files (0.0%)
       0 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 0/0/0
  257937 blocks used (0.97%)
       0 bad blocks
       1 large file

       0 regular files
       3 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
       0 symbolic links (0 fast symbolic links)
       0 sockets
--------
       3 files
root@rescue:~# fsck -yvvvvf /dev/md2
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

      12 inodes used (0.00%)
       0 non-contiguous files (0.0%)
       0 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 0/0/0
  257937 blocks used (0.97%)
       0 bad blocks
       1 large file

       0 regular files
       3 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
       0 symbolic links (0 fast symbolic links)
       0 sockets
--------
       3 files
root@rescue:~# fsck -yvvvvf /dev/md1
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

  293166 inodes used (45.66%)
    2318 non-contiguous files (0.8%)
      98 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 9776/113/0
 2391131 blocks used (93.41%)
       0 bad blocks
       2 large files

  236606 regular files
   53806 directories
      15 character device files
      77 block device files
       2 fifos
    2973 links
    2625 symbolic links (2625 fast symbolic links)
      26 sockets
--------
  296130 files
root@rescue:~# e2fsck -yvvvvf /dev/md1
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

  293166 inodes used (45.66%)
    2318 non-contiguous files (0.8%)
      98 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 9776/113/0
 2391131 blocks used (93.41%)
       0 bad blocks
       2 large files

  236606 regular files
   53806 directories
      15 character device files
      77 block device files
       2 fifos
    2973 links
    2625 symbolic links (2625 fast symbolic links)
      26 sockets
--------
  296130 files
root@rescue:~# e2fsck -yvvvvf /dev/md2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

      12 inodes used (0.00%)
       0 non-contiguous files (0.0%)
       0 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 0/0/0
  257937 blocks used (0.97%)
       0 bad blocks
       1 large file

       0 regular files
       3 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
       0 symbolic links (0 fast symbolic links)
       0 sockets
--------
       3 files
root@rescue:~# reboot

But when i restart, I just login, and then the same:

 df -h
S.ficheros            Size  Used Avail Use% Montado en
rootfs                9,7G  9,1G  158M  99% /
/dev/root             9,7G  9,1G  158M  99% /
none                  7,8G  404K  7,8G   1% /dev
/dev/md2              101G  188M   96G   1% /home
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/var/named
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.conf
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.rfc1912.zones
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/rndc.key
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/usr/lib64/bind
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.iscdlv.key
/dev/root             9,7G  9,1G  158M  99% /var/named/chroot/etc/named.root.key

Please, any idea what's going on there, I already delete the tmps from the root, and var, delete messages log... which are unsignificant, but it's unconsistency in the file system, any idea??

share|improve this question
    
Well, you can run du -sxh * from / to see which of the top level directories are taking the most space, and then drill down to the next level, etc. –  cjc Jul 12 '12 at 11:14

2 Answers 2

Find what's taking up the space with du -hx --max-depth=1 /, then repeat as necessary (du -hx --max-depth=1 /usr, for instance) until you find something big you can afford to lose or bind-mount into /home. And next time, use LVM, and don't pre-allocate all your space onto a partition you barely use.

share|improve this answer

To find out what is filling your root filesystem

cd /
du -sx | sort -rn

will give you a list of files/directories sorted by usage. Then cd into the one with the highest usage and repeat till you find out what's filling the disk.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.