The result of the df command :
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 864000688 809338092 10773908 99% /
tmpfs 32965940 0 32965940 0% /dev/shm
/dev/sda1 198337 87394 100703 47% /boot
the Result of the df command a few seconds later :
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 864000688 809400076 10711924 99% /
tmpfs 32965940 0 32965940 0% /dev/shm
/dev/sda1 198337 87394 100703 47% /boot
I'm loosing disk space on my server with an incredible speed. And everytime the available space reaches 0 octect, the Mysql service crash.
I already have reboot
the server to clear the dmesg
logs, I have deleted every big logs files (error logs, message logs and named.run logs) and run this request : sudo /usr/sbin/lsof | grep deleted
.
However the space I freed was eaten in a day time.
So I have run requests like :
du -hsx * | sort -rh | head -15
and this :
find . -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {
In every directories from my root directory.
But the biggest file I could find was a dump from my database and it's only a 1Gb file :
-rw-r--r-- 1 root root 1032237681 6 déc. 2013 dump_experta2.sql
-rw-r--r-- 1 root root 389789251 6 déc. 2013 dump_experta.sql
Can you explain me why the available space seems to be used by some invisible files ? And how may I localize and stop this space disk leak ?
Edit : Thanks to HBruijn's tips, I solved my problem, here is what I did step by step if you experience the same issue :
I ran find / -mindepth 1 -maxdepth 1 -print0 |xargs -0 du -s
And got this result :
1216308 /usr
23356 /lib64
73176 /root
440 /tmp
4 /selinux
4 /command
5574820 /var
4 /media
16 /lost+found
352388 /lib
8 /opt
0 /.autorelabel
4 /service
0 /sys
36500 /www
0 /proc
81785 /boot
7760 /bin
160 /dev
742949400 /home
14572 /sbin
0 /currentsize
0 /.autofsck
2092 /package
4 /srv
4 /mnt
27676 /etc
72944 /test
So it seems that the home directory is the biggest one so I repeated the process to find the biggest directory inside home :
# find /home -mindepth 1 -maxdepth 1 -print0 |xargs -0 du -s
133952 /home/advisio
16 /home/dovecot
2210824 /home/vpopmail
186500 /home/admin
37152 /home/ginger
511121816 /home/user1
229278612 /home/user2
And I repeated the process and saw that the user1 data has grown so I continued searching in user1 directory and found those two heavy directories :
281766156 /home/user1/www/log
207269420 /home/user1/www/fichiers
The size of the "fichiers" dir is not unusual because it is used to store countless pdf and pictures files. So I look upon the log folder and found this :
# ls -l
-rwxrwxrwx 1 vroom users 288586156425 2 juil. 15:34 log-sql-error.txt
I realised that what you said about my searching method was incredibly right. I manage to miss a 288 Gb log file by searching logs in wrong directories.
Anyway I have stoped the httpd
and the mysqld
services, emptied the error log and here is my df now :
Filesystem 1G-blocks Used Available Use% Mounted on
/dev/sda2 824G 505G 278G 65% /
tmpfs 32G 0G 32G 0% /dev/shm
/dev/sda1 1G 1G 1G 47% /boot
For now, I'll use your advices when I'll search heavy files and folders, Thank you !