Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have deployed a multi-tier java application and from time to time am seeing jobs in the application fail with broken pipe and closed channel errors similar to:

Caused by: java.nio.channels.ClosedChannelException
    at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
    at com.common.connectionmanager.AsyncSocket$AsyncSocketOutputStream.write(AsyncSocket.java:466)
    at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)

and

2009-06-15 14:02:53,492 ERROR [com.ClientConnectionManager] [Socket (ClientConnectionManager.java:368) - Client connection error 
com.ConnectionException: Error encoding message to stream
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
    at sun.nio.ch.IOUtil.write(IOUtil.java:75)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:302)
    at com.common.connectionmanager.AsyncSocket$AsyncSocketOutputStream.write(AsyncSocket.java:466)
    at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)

I think this means that the network connection between two of my servers is dropping momentarily and that these errors are telling me that the servers can't talk to each other.

All the servers are connected on a 1GBPS LAN.

The servers concerned are running Solaris - can anybody recommend any tools which I could use to monitor the network connection between them and spot whether it is indeed dropping out from time to time?

share|improve this question

3 Answers 3

I can't offer any Solaris-specific advice, but rather than testing the connection by pinging or something similar, you're probably better off monitoring the interfaces; if the connection is being dropped, I'd imagine you'd see something about it in the logs.

share|improve this answer

Run a ping test for a reasonable period of time.

ping 10.1.1.10 > connection.txt

The result will be at the end of the file connection.txt.

Ping statistics for 10.1.1.10:
Packets: Sent = 53, Received = 53, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

share|improve this answer
    
What should I learn from the result? –  Tom Melluish Jun 16 '09 at 12:39
    
If you lose some packets it means your connection isn't stable. You should get 100% received packets (or close to 99.9%) if your servers are on the same network. –  onesysadmin Jun 16 '09 at 12:45
    
Thanks - I will try to check for missing packets as you suggest –  Tom Melluish Jun 16 '09 at 13:21

You didn't mention if how these servers are connected ... LAN / WAN etc, which will make a huge difference in where you should start troubleshooting. If it's a LAN, start monitoring the switch interfaces these servers are plugged into, look for errors (maybe your duplex/speed settings are wrong), maybe one of the servers keeps requesting auto negotiation, or there's just a bad cable somewhere in the mix. You could also be seeing timeouts due to plain old congestion - there's a malware infected PC on the switch / you're running on 10BaseT.

If you're going across a WAN, start with a tool like smokeping, which will monitor a connection and tell you when, and for how long, you lose connectivity. If you are seeing drops, this could be congestion / crappy connectivity / someone is stepping on the cable at your ISP ... could be anything.

Bonne chance.

share|improve this answer
    
Thanks - all servers are connected on a gigabit LAN. Would smokeping still help? –  Tom Melluish Jun 16 '09 at 13:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.