0

I have a problem with my Ubuntu 14.04 webserver. Here in my company we are developing an app and this app accesses the webserver by sending HTTP requests. The problem is, when many PCs or devices are somehow connected to the webserver we have strange randomly timeouts here in our LAN. We are here behind a router / NAT. Accessing the webserver by 4G is absolutely fine, no timeouts, never. What did help a lot was setting net.ipv4.tcp_timestamps = 0 in /etc/sysctl.conf, this reduced the timeouts very much. But sometimes there are still timeouts and/or a slow connection to the server. I think the slowness comes from lost packets or packets which couldn't be delivered correctly, I don't know.

The server has 32gb RAM, Intel Xeon D-1540 (8 cores). It is running NginX 1.10 and PHP-FPM 7.1

Has somebody a hint for a special setting or what I can try to find a solution?

Here is my complete sysctl.conf

fs.file-max = 2097152

# Do less swapping
vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2

### GENERAL NETWORK SECURITY OPTIONS ###

# Number of times SYNACKs for passive TCP connection.
net.ipv4.tcp_synack_retries = 2

# Allowed local port range
net.ipv4.ip_local_port_range = 2000 65535

# Protect Against TCP Time-Wait
net.ipv4.tcp_rfc1337 = 1

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for connections to keep alive
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

### TUNING NETWORK PERFORMANCE ###

#Helped a lot by reducing the timeouts
net.ipv4.tcp_timestamps = 0

# Default Socket Receive Buffer
net.core.rmem_default = 31457280

# Maximum Socket Receive Buffer
net.core.rmem_max = 12582912

# Default Socket Send Buffer
net.core.wmem_default = 31457280

# Maximum Socket Send Buffer
net.core.wmem_max = 12582912

# Increase number of incoming connections
net.core.somaxconn = 50000

# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 65536

# Increase the maximum amount of option memory buffers
net.core.optmem_max = 25165824

# Increase the maximum total buffer-space allocatable
# This is measured in units of pages (4096 bytes)
net.ipv4.tcp_mem = 65536 131072 262144
net.ipv4.udp_mem = 65536 131072 262144

# Increase the read-buffer space allocatable
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.udp_rmem_min = 16384

# Increase the write-buffer-space allocatable
net.ipv4.tcp_wmem = 8192 65536 16777216
net.ipv4.udp_wmem_min = 16384

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1

# Disable TCP slow start on idle connections
net.ipv4.tcp_slow_start_after_idle = 0

NginX config (extract):

events {
        worker_connections 4096;
        multi_accept on;
        use epoll;
}

http {

        client_body_buffer_size 10K;
        client_header_buffer_size 1k;
        client_max_body_size 8m;
        large_client_header_buffers 4 16k;

        client_body_timeout 12;
        client_header_timeout 12;
        keepalive_timeout 0;
        send_timeout 10;

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        types_hash_max_size 2048;

}
  • what do you mean by "many PCs or devices" - around 10, 100, 500, 1000 or even more? can you get the output of netstat -tapen | grep TIME_WAIT | wc -l while the timeouts are occuring? – Phillip -Zyan K Lee- Stockmann Aug 19 '16 at 14:37
  • Not so many, it's already when there are 2-3 devices and some browsers who are somehow connected to the server (phpmyadmin, ssh, FTP). The output of your command is 201 right now, but there are many connections inside the server because the webserver connects over TCP to the PHP process. There isn't really much traffic on the server, 1-5 requests per second, but we are awaiting much more traffic soon. I worry that maybe a network tweak in my sysctl.conf is the reason of this behaviour. I got them from a blog article to prepare the server for higher loads. – Lumpy Aug 19 '16 at 16:06
  • It is really strange, sometimes we don't have problems, sometimes more or less. But what I see everytime is that the requests are loading slower in our LAN than using 4G connection to the server, although our LAN connection is superfast. Ping to Webserver is < 20ms. 4G Ping is > 75ms but is loading requests way faster. – Lumpy Aug 19 '16 at 16:20
  • 1
    Are packets on your lan fragmented? This may be due to some router or switch operating with wrong MTU. – Phillip -Zyan K Lee- Stockmann Aug 19 '16 at 16:26
0

If you have multiple users connecting to this webserver from behind a NAT device, having net.ipv4.tcp_tw_recycle = 1 will be causing you issues.

Please see here for a detailed description of the issue.

When the remote host is in fact a NAT device, the condition on timestamps will forbid all of the hosts except one behind the NAT device to connect during one minute because they do not share the same timestamp clock. In doubt, this is far better to disable this option since it leads to difficult to detect and difficult to diagnose problems.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.