0

We are facing an issue in our production system, running on Cent OS 7. The / (root) partition reached 90% of disk space but we don’t have any huge files / hidden files on that machine. Kindly help us to solve this issue.

For your information:

Version              :               CentOS Linux release 7.2.1511 (Core).
Machine              :               Virtual Machine.

Output of df -h

[root@dbtestlab ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G   45G  5.6G  89% /
devtmpfs                  16G     0   16G   0% /dev
tmpfs                     16G   96K   16G   1% /dev/shm
tmpfs                     16G   49M   16G   1% /run
tmpfs                     16G     0   16G   0% /sys/fs/cgroup
/dev/sdb                 493G  146G  322G  32% /logdata
/dev/sdc                 493G  181G  287G  39% /dbdata
/dev/mapper/centos-home  442G  212M  442G   1% /home
/dev/sda1                497M  195M  303M  40% /boot
tmpfs                    3.2G   16K  3.2G   1% /run/user/1000
tmpfs                    3.2G     0  3.2G   0% /run/user/0

Outpu of du -sch *

[root@dbtestlab /]# du -sch *
0       backupdata
0       bin
169M    boot
181G    dbdata
96K     dev
0       disk1
0       disk2
30M     etc
179M    home
0       lib
0       lib64
146G    logdata
0       media
0       mnt
0       opt
du: cannot access ‘proc/13270/task/13270/fd/4’: No such file or directory
du: cannot access ‘proc/13270/task/13270/fdinfo/4’: No such file or directory
du: cannot access ‘proc/13270/fd/4’: No such file or directory
du: cannot access ‘proc/13270/fdinfo/4’: No such file or directory
0       proc
40M     root
du: cannot access ‘run/user/1000/gvfs’: Permission denied
49M     run
0       sbin
0       srv
0       sys
8.0K    tmp
3.8G    usr
2.4G    var
333G    total

Partition Information:

[root@dbtestlab /]# cat /proc/partitions
major minor  #blocks  name

   2        0          4 fd0
   8       16  524288000 sdb
   8        0  524288000 sda
   8        1     512000 sda1
   8        2  523774976 sda2
   8       32  524288000 sdc
   8       48   52428800 sdd
  11        0    1048575 sr0
 253        0    8273920 dm-0
 253        1   52428800 dm-1
 253        2  463069184 dm-2

Disk Information

[root@dbtestlab /]# fdisk -l

Disk /dev/sdb: 536.9 GB, 536870912000 bytes, 1048576000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 536.9 GB, 536870912000 bytes, 1048576000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000daca1

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
/dev/sda2         1026048  1048575999   523774976   8e  Linux LVM

Disk /dev/sdc: 536.9 GB, 536870912000 bytes, 1048576000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdd: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-swap: 8472 MB, 8472494080 bytes, 16547840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-home: 474.2 GB, 474182844416 bytes, 926138368 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

EDIT 1

Inode Usage:

[root@dbtestlab /]# df -i
Filesystem                 Inodes  IUsed     IFree IUse% Mounted on
/dev/mapper/centos-root  23556176 140689  23415487    1% /
devtmpfs                  4093940    408   4093532    1% /dev
tmpfs                     4097775      9   4097766    1% /dev/shm
tmpfs                     4097775    596   4097179    1% /run
tmpfs                     4097775     13   4097762    1% /sys/fs/cgroup
/dev/sdb                 32768000     37  32767963    1% /logdata
/dev/sdc                 32768000   6612  32761388    1% /dbdata
/dev/mapper/centos-home 463069184    207 463068977    1% /home
/dev/sda1                  512000    337    511663    1% /boot
tmpfs                     4097775     26   4097749    1% /run/user/1000
tmpfs                     4097775      1   4097774    1% /run/user/0

EDIT 2

Files and folders greater than 1 GB

[root@dbtestlab /]# du -xh / 2> /dev/null | grep '[0-9].[0-9]G'
2.1G    /var/log
2.4G    /var
1.1G    /usr/lib64
1.3G    /usr/share
3.8G    /usr
6.1G    /

Kernel List:

[root@dbtestlab /]# rpm -q kernel
kernel-3.10.0-123.el7.x86_64
kernel-3.10.0-327.10.1.el7.x86_64
kernel-3.10.0-327.13.1.el7.x86_64

The kernel I am using:

[root@dbtestlab /]# uname -a
Linux dbtestlab 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I have already installed 50 GB hard disk to that machine but not mounted, since increasing size of the LVM may corrupt the system, since it is root partition. Sorry for the long question, I was searching solution for this issues and I found that this information needed for debugging, so I added all the output status. Feel free to ask me if any other further information required.

EDIT 3

Output of ls -la:

[root@dbtestlab /]# ls -la
total 52
dr-xr-xr-x.  22 root  root  4096 May 30 15:28 .
dr-xr-xr-x.  22 root  root  4096 May 30 15:28 ..
-rw-r--r--    1 root  root     0 Apr  7 12:36 .autorelabel
drwxr-xr-x    2 root  root     6 Apr 26 12:44 backupdata
lrwxrwxrwx.   1 root  root     7 Mar 22 16:56 bin -> usr/bin
dr-xr-xr-x.   4 root  root  4096 May 30 15:35 boot
drwxrwxrwx    4 mysql mysql 4096 Apr  8 16:49 dbdata
drwxr-xr-x   19 root  root  3320 May 26 23:56 dev
drwxr-xr-x.   2 root  root     6 Apr  5 23:22 disk1
drwxr-xr-x.   2 root  root     6 Apr  5 23:22 disk2
drwxr-xr-x. 141 root  root  8192 May 30 15:33 etc
drwxr-xr-x.   6 root  root  4096 May 30 15:59 home
lrwxrwxrwx.   1 root  root     7 Mar 22 16:56 lib -> usr/lib
lrwxrwxrwx.   1 root  root     9 Mar 22 16:56 lib64 -> usr/lib64
drwxrwxrwx    8 mysql mysql 4096 Apr  9 22:59 logdata
drwxr-xr-x.   2 root  root     6 Aug 12  2015 media
drwxr-xr-x.   2 root  root     6 Aug 12  2015 mnt
drwxr-xr-x.   3 root  root    15 Aug 12  2015 opt
dr-xr-xr-x  250 root  root     0 May 26 23:53 proc
dr-xr-x---.  14 root  root  4096 May 30 14:37 root
drwxr-xr-x   38 root  root  1180 May 30 17:23 run
lrwxrwxrwx.   1 root  root     8 Mar 22 16:56 sbin -> usr/sbin
drwxr-xr-x.   2 root  root     6 Aug 12  2015 srv
dr-xr-xr-x   13 root  root     0 May 26 23:53 sys
drwxrwxrwt.  14 root  root  4096 May 31 03:28 tmp
drwxr-xr-x.  13 root  root  4096 Mar 22 16:56 usr
drwxr-xr-x.  23 root  root  4096 May 26 23:53 var

Output of cat /proc/swaps

[root@dbtestlab /]# cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/dm-0                               partition       8273916 899852  -1

Output of pvdisplay

[root@dbtestlab /]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               centos
  PV Size               499.51 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              127874
  Free PE               0
  Allocated PE          127874
  PV UUID               KP0RnD-kAZ3-sJ0o-B7SH-ePI3-8zig-2Vhmq3

Output of du -sch /* --exclude=/{dev,run,sys,logdata,dbdata,home,boot}

[root@dbtestlab /]# du -sch /* --exclude=/{dev,run,sys,logdata,dbdata,home,boot}
0       /backupdata
0       /bin
0       /disk1
0       /disk2
30M     /etc
0       /lib
0       /lib64
0       /media
0       /mnt
0       /opt
du: cannot access ‘/proc/13991/task/13991/fd/4’: No such file or directory
du: cannot access ‘/proc/13991/task/13991/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/13991/fd/4’: No such file or directory
du: cannot access ‘/proc/13991/fdinfo/4’: No such file or directory
0       /proc
40M     /root
0       /sbin
0       /srv
8.0K    /tmp
3.8G    /usr
2.4G    /var
6.1G    total
Improve this question
7
  • 2
    I'd run lsof and check if there are files which are opened but marked for deletion, if so killing their PID should free some more space and could even solve your issue.
    – Itai Ganot
    May 30, 2016 at 15:21
  • Database temp tables?
    – HTTP500
    May 30, 2016 at 15:23
  • can you provide output of "ls -la /", "cat /proc/swaps" and "pvdisplay"?
    – maxxvw
    May 30, 2016 at 15:36
  • Please take a look at this Q&A
    – HBruijn
    May 30, 2016 at 15:37
  • Probably you have deleted files as Itai Ganot mentions: run lsof -n | grep deleted with root permissions to see if you have any open files that have been marked for deletion but are still using the disk space.
    – Pablo Martinez
    May 30, 2016 at 18:03

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .