I have a webapp running on some servers that have their Apache log files on an OCFS2 SAN volume. The internal webapp logging is done using log4perl, in pipe mode, with cronolog on the other end of the pipe to handle log file rotation. From time to time, right around midnight, when the log files are rotated, some of the servers will experience a broken pipe, which renders Apache unable to log, and generates all sorts of alerts and bad things.
I'm wondering if there is a way to determine the root cause of the pipe breaking. DTrace sounds like it could do it, with some clever programming, but I'm not sure.
Anyone have any ideas?