0

I have deployed a multi-tier java application and from time to time am seeing jobs in the application fail with broken pipe and closed channel errors similar to:

Caused by: java.nio.channels.ClosedChannelException
    at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
    at com.common.connectionmanager.AsyncSocket$AsyncSocketOutputStream.write(AsyncSocket.java:466)
    at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)

and

2009-06-15 14:02:53,492 ERROR [com.ClientConnectionManager] [Socket (ClientConnectionManager.java:368) - Client connection error 
com.ConnectionException: Error encoding message to stream
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
    at sun.nio.ch.IOUtil.write(IOUtil.java:75)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:302)
    at com.common.connectionmanager.AsyncSocket$AsyncSocketOutputStream.write(AsyncSocket.java:466)
    at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)

I think this means that the network connection between two of my servers is dropping momentarily and that these errors are telling me that the servers can't talk to each other.

All the servers are connected on a 1GBPS LAN.

The servers concerned are running Solaris - can anybody recommend any tools which I could use to monitor the network connection between them and spot whether it is indeed dropping out from time to time?

Improve this question

3 Answers 3

Reset to default
1

Run a ping test for a reasonable period of time.

ping 10.1.1.10 > connection.txt

The result will be at the end of the file connection.txt.

Ping statistics for 10.1.1.10:
Packets: Sent = 53, Received = 53, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

Improve this answer
3
  • What should I learn from the result?
    – Tom Melluish
    Jun 16, 2009 at 12:39
  • If you lose some packets it means your connection isn't stable. You should get 100% received packets (or close to 99.9%) if your servers are on the same network.
    – Jindrich
    Jun 16, 2009 at 12:45
  • Thanks - I will try to check for missing packets as you suggest
    – Tom Melluish
    Jun 16, 2009 at 13:21
1

You didn't mention if how these servers are connected ... LAN / WAN etc, which will make a huge difference in where you should start troubleshooting. If it's a LAN, start monitoring the switch interfaces these servers are plugged into, look for errors (maybe your duplex/speed settings are wrong), maybe one of the servers keeps requesting auto negotiation, or there's just a bad cable somewhere in the mix. You could also be seeing timeouts due to plain old congestion - there's a malware infected PC on the switch / you're running on 10BaseT.

If you're going across a WAN, start with a tool like smokeping, which will monitor a connection and tell you when, and for how long, you lose connectivity. If you are seeing drops, this could be congestion / crappy connectivity / someone is stepping on the cable at your ISP ... could be anything.

Bonne chance.

Improve this answer
1
  • Thanks - all servers are connected on a gigabit LAN. Would smokeping still help?
    – Tom Melluish
    Jun 16, 2009 at 13:20
0

I can't offer any Solaris-specific advice, but rather than testing the connection by pinging or something similar, you're probably better off monitoring the interfaces; if the connection is being dropped, I'd imagine you'd see something about it in the logs.

Improve this answer

You must log in to answer this question.