Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've got a box where the root partition is full.

The most immediate explanation is the mysql binary logs aren't rotating. Despite expire_log_days being set to 10, there are logs older than 10 days. show master status, show slave status, and purge binary logs all hang indefinitely.

share|improve this question
add comment

3 Answers 3

up vote 0 down vote accepted

I'd try a few things, but make sure / being full is your problem. Normally Mysql stores data in /var/lib/mysql and doesn't use / for much other than /tmp.

First try to clear some space. Delete old logs from /var/log/ or whack that src code you downloaded and never used. If you can get / down to less than 100% you Mysql might return to a state where it's not wedged.

Assuming you've cleared some space try to purge logs. I'd try PURGE BINARY LOGS BEFORE DATE_SUB( NOW( ), INTERVAL 3 DAY); from the Mysql commandline. If it still hangs after ten minutes you'll probably need to shut Mysql down if can or kill it if you can't.

Normally it's never a good idea to delete logs manually, but in this case it might be your only choice. Delete the oldest log and try to restart Mysql. It'll probably have to do some table checks. Keep an eye on the mysql error log until it's up and running properly. Then try to purge logs again.

Once it's up and running make sure expire_logs_days is really set by checking show variables; in Mysql. You may also want to set max_binlog_size = 100M if your filesystem is small. Normally Mysql rotates logs at 1G which might not be often enough. Mysql only rotates logs when the current log reaches max size.

share|improve this answer
    
/var/lib is on the root partition in this box –  ʞɔıu Mar 16 '11 at 17:32
add comment

This depends entirely on the version of MySQL you are running

The problem lies with the memory map of of all binary logs.

When mysqld starts, it reads the index of mysql binary logs into memory.

That memory map is sensitive to the presence of binary logs on disk.

If you do commands such as:
1. PURGE BINARY LOGS TO '< binary log filename >';
2. PURGE BINARY LOGS BEFORE '< date and time >';
3. RESET MASTER;

mysqld will use the memory map instead of the binary log index file.

If the memory map does not precisely match the binary logs files on disk (i.e., not in sych), log rotation with expire_logs_days will fail.

You cannot simply edit the mysql binary log index file, because when you issue a shutdown for mysqld, the memory map will overwrite the binary log index file.

The problem usually happens when someone deletes the binary logs in the operating system rather that using one of the aformentioned three(3) MySQL Commands. mysqld will not just pick up on the sudden disappearance of binary logs.

The only immediate correction possible is to delete all binary logs, delete the binary log index file, and start mysql back up, or you could do the following:

  1. shutdown mysql
  2. delete binary logs you you do not want
  3. edit the binary log index file (remove binary logs that do not exist from the file)
  4. start mysqld

To watch out for disappearing binary logs thereafter, run SHOW BINARY LOGS; if any of the binary logs entries comes back with file size 0, the memory map of binary logs is out of sych. The course of action then is to do the following:

  1. shutdown mysql
  2. edit the binary log index file (remove binary logs that do not exist from the file)
  3. start mysqld
share|improve this answer
add comment

My advice is once you get it rescued, put /var/lib/mysql and /var/log/mysql on their own LVM volumes.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.