centos 100% disk full - How to remove log files, history, etc?

Question

mysqld won't start because disk space is full:

101221 14:06:50 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
101221 14:06:50 [ERROR] Can't start server: can't create PID file: No space left on device

running df -h:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              16G  3.2G   12G  23% /
/dev/sda5             4.8G  4.6G     0 100% /var
/dev/sda3             430G  855M  407G   1% /home
/dev/sda1              76M   24M   49M  33% /boot
tmpfs                 956M     0  956M   0% /dev/shm

du -sh * in /var:

12K     account
56M     cache
24K     db
32K     empty
8.0K    games
1.5G    lib
8.0K    local
32K     lock
221M    log
16K     lost+found
0       mail
24K     named
8.0K    nis
8.0K    opt
8.0K    preserve
8.0K    racoon
292K    run
70M     spool
8.0K    tmp
76K     webmin
2.6G    www
20K     yp

in /dev/sda5, there is website files in /var/www.

because this is first time, I have no idea which files to remove other than moving /var/www to other partition

And one more, what is the right way to remove log files, history, etc in /dev/sda5?

Answer 1

Quick and dirty: Move your /var/www to /home/www, then make a symlink:

rsync -a /var/www /home &&
rm -r /var/www &&
ln -s /home/www /var/www

Longer term: Maybe put your /var/www and /var/log directories into their own dedicated partitions. You could probably shrink /home and use the space freed up to create suitable partitions.

Answer 2

1. I would mv /var/log/*gz /var/log/*.0 /home/backup/var/log
2. yum clean packages or apt-get clean to remove stuff from /var/cache
3. Yes move /var/www to e.g. /home
4. perhaps remove the /var partition..

Answer 3

What's the breakdown on /var/lib? If it's like my system, most of that (say 500 MB) is taken up by /var/lib/mysql?

The other numbers all look pretty normal and reasonable, so moving /var/lib/mysql and /var/www are the only realistic long-term solutions.

For removing old log files, are your logs being rotated, e.g. do you have /var/log/messages, /var/log/messages.1, /var/log/messages.2.gz, etc, or maybe /var/log/messages-20101221, /var/log/messages-20101220.gz, etc?

The obvious way to remove those is by age, e.g.

# find /var/log -type f -mtime +14 -print
# find /var/log -type f -mtime +14 -exec rm '{}' \;

Also, check that your logrotate settings in /etc/logrotate.conf are what you want. I'd say you shouldn't change them, as they're not your main problem, but maybe you only want to keep a week's logs, in which case try something like this:

daily
rotate 7
create
compress
include /etc/logrotate.d

Then there are files for each service, e.g. syslog, mysql, apache, etc to configure log rotation for each of those. On Debian and Red Hat systems, that should already be set up for you.

If you don't already have logrotate, then it's a bit harder. I'd look at using grep or tail to save the recent entries then deleting the file and restarting syslogd (e.g. service syslogd restart or pkill -HUP syslogd.

Finally, a script I wrote for just this situation is diskuse.

# test -d ~/bin || mkdir ~/bin
# svn cat http://svn.mikelward.com/svn/scripts/diskuse > ~/bin/diskuse
# chmod +x ~/bin/diskuse
# ~/bin/diskuse -A /var > ~/biggest-files-in-var.txt
# head ~/biggest-files-in-var.txt
# ~/bin/diskuse -T -A /var > ~/biggest-dirs-in-var.txt
# head ~/biggest-dirs-in-var.txt
# ~/bin/diskuse -a 1d /var > ~/biggest-files-in-var-that-changed-today.txt
# head ~/biggest-files-in-var-that-changed-today.txt