I have a Nagios alert about disk space on our
/var/log mountpoint on a production system.
We have a
catalina.out (Tomcat) file which has grown to 768 megs, and there are 171 megs left on the mount point.
Obviously logrotate should have been running on this machine, but wasn't. The standard practice for
catalina.out is to use
copytruncate to be able to process
catalina.out without bringing down Tomcat in order to do it.
If I simply add our standard Tomcat logrotate configuration, which uses
copytruncate, I assume the copy happens before the truncate, and therefore I'll run out of disk space.
The log stretches back to December last year, so it's not essential to keep all of the log's contents.
What's the safest way to get out of this situation? Purely logically speaking, I think I need to get rid of the beginning 500 megs of this file before I install the logrotate configuration, but I don't even know if that's possible.
I'm running RHEL 5.3.