mysqld won't start because disk space is full:

101221 14:06:50 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
101221 14:06:50 [ERROR] Can't start server: can't create PID file: No space left on device

running df -h:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              16G  3.2G   12G  23% /
/dev/sda5             4.8G  4.6G     0 100% /var
/dev/sda3             430G  855M  407G   1% /home
/dev/sda1              76M   24M   49M  33% /boot
tmpfs                 956M     0  956M   0% /dev/shm

du -sh * in /var:

12K     account
56M     cache
24K     db
32K     empty
8.0K    games
1.5G    lib
8.0K    local
32K     lock
221M    log
16K     lost+found
0       mail
24K     named
8.0K    nis
8.0K    opt
8.0K    preserve
8.0K    racoon
292K    run
70M     spool
8.0K    tmp
76K     webmin
2.6G    www
20K     yp

in /dev/sda5, there is website files in /var/www.

because this is first time, I have no idea which files to remove other than moving /var/www to other partition

And one more, what is the right way to remove log files, history, etc in /dev/sda5?


Quick and dirty: Move your /var/www to /home/www, then make a symlink:

rsync -a /var/www /home &&
rm -r /var/www &&
ln -s /home/www /var/www

Longer term: Maybe put your /var/www and /var/log directories into their own dedicated partitions. You could probably shrink /home and use the space freed up to create suitable partitions.

  1. I would mv /var/log/*gz /var/log/*.0 /home/backup/var/log
  2. yum clean packages or apt-get clean to remove stuff from /var/cache
  3. Yes move /var/www to e.g. /home
  4. perhaps remove the /var partition..
  • FWIW: I don't really like using partition when there is no real use for them. If you want to use them as administrative boundaries you better have LVM/ZFS or some kind of easy way to extend them.. Dec 22 '10 at 6:54
  • I don't really like using partition too...
    – kopeklan
    Dec 22 '10 at 10:30

What's the breakdown on /var/lib? If it's like my system, most of that (say 500 MB) is taken up by /var/lib/mysql?

The other numbers all look pretty normal and reasonable, so moving /var/lib/mysql and /var/www are the only realistic long-term solutions.

For removing old log files, are your logs being rotated, e.g. do you have /var/log/messages, /var/log/messages.1, /var/log/messages.2.gz, etc, or maybe /var/log/messages-20101221, /var/log/messages-20101220.gz, etc?

The obvious way to remove those is by age, e.g.

# find /var/log -type f -mtime +14 -print
# find /var/log -type f -mtime +14 -exec rm '{}' \;

Also, check that your logrotate settings in /etc/logrotate.conf are what you want. I'd say you shouldn't change them, as they're not your main problem, but maybe you only want to keep a week's logs, in which case try something like this:

rotate 7
include /etc/logrotate.d

Then there are files for each service, e.g. syslog, mysql, apache, etc to configure log rotation for each of those. On Debian and Red Hat systems, that should already be set up for you.

If you don't already have logrotate, then it's a bit harder. I'd look at using grep or tail to save the recent entries then deleting the file and restarting syslogd (e.g. service syslogd restart or pkill -HUP syslogd.

Finally, a script I wrote for just this situation is diskuse.

# test -d ~/bin || mkdir ~/bin
# svn cat http://svn.mikelward.com/svn/scripts/diskuse > ~/bin/diskuse
# chmod +x ~/bin/diskuse
# ~/bin/diskuse -A /var > ~/biggest-files-in-var.txt
# head ~/biggest-files-in-var.txt
# ~/bin/diskuse -T -A /var > ~/biggest-dirs-in-var.txt
# head ~/biggest-dirs-in-var.txt
# ~/bin/diskuse -a 1d /var > ~/biggest-files-in-var-that-changed-today.txt
# head ~/biggest-files-in-var-that-changed-today.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.