I have deployed a multi-tier java application and from time to time am seeing jobs in the application fail with broken pipe and closed channel errors similar to:
Caused by: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294) at com.common.connectionmanager.AsyncSocket$AsyncSocketOutputStream.write(AsyncSocket.java:466) at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)
2009-06-15 14:02:53,492 ERROR [com.ClientConnectionManager] [Socket (ClientConnectionManager.java:368) - Client connection error com.ConnectionException: Error encoding message to stream Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:302) at com.common.connectionmanager.AsyncSocket$AsyncSocketOutputStream.write(AsyncSocket.java:466) at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)
I think this means that the network connection between two of my servers is dropping momentarily and that these errors are telling me that the servers can't talk to each other.
All the servers are connected on a 1GBPS LAN.
The servers concerned are running Solaris - can anybody recommend any tools which I could use to monitor the network connection between them and spot whether it is indeed dropping out from time to time?