3

A quite busy proxy server has lots of "SYNs to LISTEN sockets dropped".

I learned one cause could be a too small backlog size. But in that case the "times the listen queue of a socket overflowed" value should be equal (which it is not).

So what could be a cause for this behaviour? Maybe a broken nic?

We have 5 proxies, in 2 of which the two numbers are not equal, so this problem seems to be happening there.

Here the output from netstat:

$ netstat -s | grep -i list
238627 times the listen queue of a socket overflowed
8610307 SYNs to LISTEN sockets dropped

the servers have ipv4 and ipv6 traffic, maybe that helps?

6

These counters ultimately come from the kernel and map to the LINUX_MIB_LISTENOVERFLOWS and LINUX_MIB_LISTENDROPS counters. You can see from the source of net/ipv4/tcp_ipv4.c(tcp_v4_syn_recv_sock) around line #1392 that when LINUX_MIB_LISTENOVERFLOWS is incremented, LINUX_MIB_LISTENDROPS will also be incremented but there are exit conditions where only the latter can be incremented so it's not a bug that they don't match.

In the same file you can see there's this code:

1291 int tcp_v4_conn_request(struct sock *sk, struct sk_buff *skb)
1292 {
1293         /* Never answer to SYNs send to broadcast or multicast */
1294         if (skb_rtable(skb)->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST))
1295                 goto drop;
1296 
1297         return tcp_conn_request(&tcp_request_sock_ops,
1298                                 &tcp_request_sock_ipv4_ops, sk, skb);
1299 
1300 drop:
1301         NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_LISTENDROPS);
1302         return 0;
1303 }

So you can see at least one cause is a SYN to a broadcast or multicast address.

  • Thanks for the explanation. From the code i see two possibilities for growing LINUX_MIB_LISTENOVERFLOWS: 1) syn to broadcast as mentioned by you 2) Still in SYN_RECV, just remove it silently. (i honestly dont understand what is meant there). As the counts diverge on 2 of 5 proxies (which are all configured in the same way), can you suggest a cause for the behaviour? – edlerd Nov 24 '14 at 12:25
  • Nothing definitive, however I would monitor the value of both counters over time and see what the general rate and pattern is. Bear in mind there is a separate yet similar code path for TCP & IPv6 so you might have non-unicast IPv6 traffic causing the counters to increase. – bodgit Nov 24 '14 at 13:32
  • i was monitoring the numbers again today. so the result is, that the SYNs to LISTEN sockets dropped is increased by ~100k, while the other counter is unchanged. Also I can see the first value change every couple of seconds. Any idea how to identify the reason? – edlerd Nov 26 '14 at 21:54
0

Usually wmem and rmem defaults are 212992 bytes. Apparently not enough on busy server. Raised to 8MB and the problem disappeared.

sysctl -w net.core.wmem_default=8388608
sysctl -w net.core.rmem_default=8388608
  • If you are already setting net.ipv4.tcp_rmem and net.ipv4.tcp_wmem and you are only serving TCP traffic, is there a point in also setting the net core values? – andresp May 13 '19 at 10:44
  • According to IBM those net.core properties are superseeded by the TCP ones for TCP connections: ibm.com/support/knowledgecenter/en/linuxonibm/liaag/wkvm/… so there is no point in setting them if you already set the ones for TCP. – andresp May 13 '19 at 10:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.