2

When you have a low disk situation with a log file taking up some GB and you have no disk space left what is the best course of action without losing any of logs?

Things I tried was to mv the log file and compress, but this will show up as deleted in lsof which might be a problem in the future.

Possibly gzip the log file and kill -1 the process. But is this something that you do a production server for services like httpd and mysql?

Thanks.
Ob

2

If storage is 100% full, the compression won't work as there is no space for a temporary file.

Copy logs to other storage. scp -r /var/log/ otherhost:

Review and delete old log files. find /var/log -mtime +7

Expand file system if necessary.

Compress some large files. Reload services to open a new log file. gzip /var/log/httpd/access_log ; systemctl reload httpd.service

Implement logrotate or equivalent script to manage these automatically. The usual pattern is to move the current file to a new name, and reopen a new log file.

Consider implementing a remote log server and shipping logs off the host instead.


Whether sending a signal to a service or otherwise reloading it is acceptable is up to you. Of course, you can try it on a test system if this makes you nervous.

If you don't tell the service to open a new file there is another option: truncate in place. cp /dev/null file.log or logrotate option copytruncate. However, beware the warning about this not being atomic from the lograte man page:

Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost.

| improve this answer | |
1

for apache use USR1 signal:

killall -USR1 httpd

and for mysql flush logs command:

mysqladmin flush-logs

but probably you would be better by using logrotate to rotate logs and usually it comes with configs for apache/mysql, which basically does it for you and gzips logs. It is common practice on production servers.

| improve this answer | |
  • 1
    This answer needs a lot more detail. A reader who doesn't know how logrotate works isn't going to know after reading this answer either. – kasperd Dec 19 '17 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.