0

The result of the df command :

Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sda2      864000688 809338092  10773908  99% /
tmpfs           32965940         0  32965940   0% /dev/shm
/dev/sda1         198337     87394    100703  47% /boot

the Result of the df command a few seconds later :

Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sda2      864000688 809400076  10711924  99% /
tmpfs           32965940         0  32965940   0% /dev/shm
/dev/sda1         198337     87394    100703  47% /boot

I'm loosing disk space on my server with an incredible speed. And everytime the available space reaches 0 octect, the Mysql service crash.

I already have reboot the server to clear the dmesg logs, I have deleted every big logs files (error logs, message logs and named.run logs) and run this request : sudo /usr/sbin/lsof | grep deleted.

However the space I freed was eaten in a day time.

So I have run requests like :

du -hsx * | sort -rh | head -15

and this :

find . -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {

In every directories from my root directory.

But the biggest file I could find was a dump from my database and it's only a 1Gb file :

-rw-r--r-- 1 root root 1032237681  6 déc.   2013 dump_experta2.sql
-rw-r--r-- 1 root root  389789251  6 déc.   2013 dump_experta.sql

Can you explain me why the available space seems to be used by some invisible files ? And how may I localize and stop this space disk leak ?

Edit : Thanks to HBruijn's tips, I solved my problem, here is what I did step by step if you experience the same issue :

I ran find / -mindepth 1 -maxdepth 1 -print0 |xargs -0 du -s

And got this result :

1216308 /usr
23356   /lib64
73176   /root
440     /tmp
4       /selinux
4       /command
5574820 /var
4       /media
16      /lost+found
352388  /lib
8       /opt
0       /.autorelabel
4       /service
0       /sys
36500   /www
0       /proc
81785   /boot
7760    /bin
160     /dev
742949400       /home
14572   /sbin
0       /currentsize
0       /.autofsck
2092    /package
4       /srv
4       /mnt
27676   /etc
72944   /test

So it seems that the home directory is the biggest one so I repeated the process to find the biggest directory inside home :

# find /home -mindepth 1 -maxdepth 1 -print0 |xargs -0 du -s
133952  /home/advisio
16      /home/dovecot
2210824 /home/vpopmail
186500  /home/admin
37152   /home/ginger
511121816       /home/user1
229278612       /home/user2

And I repeated the process and saw that the user1 data has grown so I continued searching in user1 directory and found those two heavy directories :

281766156       /home/user1/www/log
207269420       /home/user1/www/fichiers

The size of the "fichiers" dir is not unusual because it is used to store countless pdf and pictures files. So I look upon the log folder and found this :

# ls -l
-rwxrwxrwx 1 vroom users 288586156425  2 juil. 15:34 log-sql-error.txt

I realised that what you said about my searching method was incredibly right. I manage to miss a 288 Gb log file by searching logs in wrong directories.

Anyway I have stoped the httpd and the mysqld services, emptied the error log and here is my df now :

Filesystem     1G-blocks  Used Available Use% Mounted on
/dev/sda2           824G  505G      278G  65% /
tmpfs                32G    0G       32G   0% /dev/shm
/dev/sda1             1G    1G        1G  47% /boot

For now, I'll use your advices when I'll search heavy files and folders, Thank you !

3

First: the sequence find . -type d -print0 | xargs -0 du | sort -n is incredibly recursive and inefficient.
Finding the largest files, with find / -type f -print0 | xargs -0 du | sort -n is already much better with find / -type f -printf "%s %p\n" |sort -n even more efficient.

Often more telling is where you lose your disk space and finding the largest directories quickly (du-s) instead because many small files also add up (think mail queue, print spool etc.)

Second: when you ran find "in every directories from my root directory" you may have overlooked "hidden" files/directories under your root / i.e. simple ones starting with a dot /.<name> and less obvious ones such as ones consisting of only a space / / or a TAB character.

Start with:

find / -mindepth 1 -maxdepth 1 -print0 |xargs -0 du -s 

and work your way down from that.

You already looked at deleted files that might still be held open with lsof but you might have processes writing large (temporary) files and it might be interesting and useful to watch sudo lsof -s | awk '$5 == "REG"' | sort -n -r -k 7,7 | head -n 50

  • 1
    you can shorten that without a pipe just using du du / -s --max-depth=1 – Mike Jul 2 '15 at 14:17
  • My Mac with the BSD version of tools always trips me up, yes indeed the GNU version of du has its own depth limiter! – HBruijn Jul 2 '15 at 14:23
  • yep... yours is more will work everywhere command – Mike Jul 2 '15 at 14:30
0

In the future, use a tool like ncdu to map your filesystem usage. You would have found the log file issue very easily.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.