1

I have a number of linux servers at a us-based datacenter. The servers were installed by the hosting company, and are running fedora core.

We're experiencing problems with dropped connections. The issue seems to be that when we attempt to connect to one of the other servers after a period of inactivity, the first connection attempt will fail, and sometimes the second. However, after that the connection succeds and it works for a while. This happens for both mysql connections and raw socket connections, but only seems to occur when connecting to some of our servers. The confusing part is that it some of the servers for which we see different behaviors have identical hardware configuration and software. For example, it happens when connecting to a server called mysql2, but not for a server called mysql3. These servers were installed at the same time, with the same specifications.

The problem can be reproduced somewhat reliably, but only after waiting fifteen minutes to half an hour. This makes it hard to diagnose, and even harder since I'm not really sure what to look for.

I realize that connections sometimes failed, and that we should write our applications to compensate for this but these servers all in the same data center. Why would it matter if two servers haven't communicated for a while?

Does anybody have an idea what might be causing this? Is it a server configuration problem or a network problem that I should contact the hosting company about. What do I tell them to look for? Unfortunately our experience has been that the support staff doesn't investigate problems in depth unless we give them detailed directions.

Edit: MySQL seems to at least receive the connections before they are dropped:

mysql> show status like 'abort%';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| Aborted_clients  | 501   |
| Aborted_connects | 184   |
+------------------+-------+
2 rows in set (0.00 sec)

Edit2: Seems like stateful firewalls was the problem after all. Flushing out iptables seemed to do the trick.

iptables -P OUTPUT ACCEPT
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -F INPUT
iptables -F OUTPUT
iptables -F FORWARD

1 Answer 1

2

Have you run tcpdump/wireshark to trace the connection behavior? My money is on some sort of state-full firewall between your two locations. Your software opens a connection that then doesn't see any traffic for a few minutes and the firewall times out the connection and subsequent data will be dropped and maybe the connection will be reset.

I would trace some of the connections in tcpdump to see how they are terminated and if there's a large amount of idle time. If it is this firewall thing, you could try and enabling tcp keep alive on the connections. I don't know how easy it will be to do that, though.

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .