Rotating Tomcat logs on Solaris with logdam creates growing block of nulls

Question

I need to rotate some Tomcat logs in Solaris. When I try to use logadm with the -c option, the rotation seems to occur correctly, but then catalina.out becomes a block of nulls the same size as the rotated part, and new data is written after these nulls. It's as if the file position pointer never moved, and Tomcat's output keeps advancing from the same position.

As a result, catalina.out continues to grow. The leading block of nulls grows with every rotation event.

When it works, this is the most convenient log rotation solution I know of, because you don't have to modify catalina.sh. But it isn't working, and the alternative -- piping the output through Apache logrotate -- requires modifying catalina.sh, which I'd rather not do. It means another local change to track every time we update Tomcat.

Answer 1

Actually, catalina.sh does intend to open catalina.out with O_APPEND: >> "$CATALINA_OUT" 2>&1 &

The root cause is the shell: #!/bin/sh. Solaris' default shell (/bin/sh) does not interpret >> correctly to open the output file with O_APPEND.

This is not a problem on Linux. You can work around it for Solaris with a standards compliant shell: /usr/xpg4/bin/sh (see man sh)

Modify catalina.sh and replace #!/bin/sh with #!/usr/xpg4/bin/sh. The disadvantage is that every time you update Tomcat, you need to redo it.

It is easy to verify whether catalina.out is opened with O_APPEND or not using the pfiles command.

1: S_IFREG mode:0644 dev:32,5 ino:13738 uid:0 gid:0 size:10170
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
   /opt/tomcat6/logs/catalina.out

You can see that /usr/xpg4/bin/sh opens catalina.out with O_APPEND.

Answer 2

It's exactly that; Tomcat (catalina.sh, to be specific) doesn't open its log file with O_APPEND, and thus writes occur at the previous position, creating a hole (nulls, taking up no disk space) up to that point.

One option is to have a post_command to restart Tomcat after the rotation.

Another is to make catalina.out a named pipe and have a program reading from it and doing "the right thing". The flaw with that is that if the program is not running, Tomcat will fill up the pipe buffer and then block until it is drained.