0

I have been running my server for the past 6 years without any problem, but recently ran into some trouble. I am getting a lot of SYN_RECV connections which is pushing the backlog of connecting devices so far back and nothing is updating as it should. Now my server is a Intel(R) Xeon(R) CPU E31230 @ 3.20GHz with 7 cores with 32GB memory and running CentOS 6.7 Red Hat. I have about +- 800 active connections running on this machine. And I have 12 ports running and all +-800 devices is split over these 12 ports with max 100 devices per port. I have my sockets created/running through PHP. I would appreciate ANY advice.

-Would it help to have everything on 1 port? -Would it help to extent them over more ports (Max 50 devices per port) -Would it help to tweak the linux settings (which settings would it be) -Would it help to write it in another language (which would be best if any) -What else could cause this problem?

I have SYN Cookies enable to help with SYN Attacks:

sysctl -n net.ipv4.tcp_syncookies

This is my sysctl.conf:

/# Kernel sysctl configuration file for Red Hat Linux /# /# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and /# sysctl.conf(5) for more details.

/# Controls IP packet forwarding net.ipv4.ip_forward = 0

/# Controls source route verification net.ipv4.conf.default.rp_filter = 1

/# Do not accept source routing net.ipv4.conf.default.accept_source_route = 0

/# Controls the System Request debugging functionality of the kernel kernel.sysrq = 0

/# Controls whether core dumps will append the PID to the core filename. /# Useful for debugging multi-threaded applications. kernel.core_uses_pid = 1

/# Controls the use of TCP syncookies net.ipv4.tcp_syncookies = 1

/# Disable netfilter on bridges. net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0

/# Controls the default maxmimum size of a mesage queue kernel.msgmnb = 65536

/# Controls the maximum size of a message, in bytes kernel.msgmax = 65536

/# Controls the maximum shared segment size, in bytes kernel.shmmax = 68719476736

/# Controls the maximum number of shared memory segments, in pages kernel.shmall = 4294967296

/# Retry SYN/ACK only three times, instead of five net.ipv4.tcp_synack_retries = 5

/# Try to close things only twice net.ipv4.tcp_orphan_retries = 5

/# FIN-WAIT-2 for only 5 seconds net.ipv4.tcp_fin_timeout = 30

/# Increase syn socket queue size (default: 512) net.ipv4.tcp_max_syn_backlog = 1024 net.core.netdev_max_backlog = 1000

/# One hour keepalive with fewer probes (default: 7200 & 9) net.ipv4.tcp_keepalive_time = 7200 net.ipv4.tcp_keepalive_probes = 5

/# Max packets the input can queue net.core.netdev_max_backlog = 65536

/# Keep fragments for 15 sec (default: 30) net.ipv4.ipfrag_time = 30

/# Use H-TCP congestion control net.ipv4.tcp_congestion_control = htcp

net.core.rmem_default = 256960 net.core.rmem_max = 256960 net.core.wmem_default = 256960 net.core.wmem_max = 256960

net.ipv4.tcp_sack = 1 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 0 net.ipv4.ip_local_port_range = 15000 61000 net.core.somaxconn = 1024

Minus the "/" before the "#"

Here is my PHP code for running the ports:

#!/usr/bin/php -q
<?php
    error_reporting(0);
    set_time_limit(0);
    ob_implicit_flush();
    $address = '123.123.123.123';
    $port = 8000;

    //Check for the connections
    if (($master = socket_create(AF_INET, SOCK_STREAM, SOL_TCP)) < 0) 
    { 
        logs("socket_create() failed, reason: " . socket_strerror($master) . "\n", $enable_logging); 
    }

    socket_set_option($master, SOL_SOCKET,SO_REUSEADDR, 1); 

    if (($ret = socket_bind($master, $address, $port)) < 0) 
    { 
        logs("socket_bind() failed, reason: " . socket_strerror($ret) . "\n", $enable_logging); 
    }

    if (($ret = socket_listen($master, SOMAXCONN)) < 0) 
    { 
        logs("socket_listen() failed, reason: " . socket_strerror($ret) . "\n", $enable_logging); 
    } 

    $read_sockets = array($master);

    //Read all data from buffer
    while (true) 
    { 
        $changed_sockets = $read_sockets; 
        $num_changed_sockets = socket_select($changed_sockets, $write = NULL, $except = NULL, NULL); 
        foreach($changed_sockets as $socket) 
        { 

            if ($socket == $master) 
            { 
                if (($client = socket_accept($master)) < 0) 
                { 
                    logs("socket_accept() failed: reason: " . socket_strerror($msgsock) . "\n", $enable_logging); 
                    continue; 
                }
                else
                { 
                    array_push($read_sockets, $client);
                    logs("[".date('Y-m-d H:i:s')."] ".$client." CONNECTED "."(".count($read_sockets)."/".SOMAXCONN.")\r\n", $enable_logging);
                } 
            }
            else
            { 
                $buffer = socket_read($socket, 8192);
                if ($buffer === "") 
                { 
                    $index = array_search($socket, $read_sockets); 
                    unset($read_sockets[$index]); 
                    socket_close($socket); 
                }
                else
                {
                    //Do DB connections etc here
                }
            }
        }
    }
?>

I open and close the ports with a bash script.

Any any help, I'm open for anything.

  • So are these SYN requests malicious or not? Can you differentiate between expected connections and rouge ones? If so, you can use iptables rate limiting, or fail2ban, or Nginx throttling to keep the server stable. Also, you may want to expand on your PHP setup, the server etc. – JayMcTee Aug 19 '15 at 12:24
  • What I can see is that the SYN requests are from my Tracking units connecting to the server. With that also being said, I do also experience lost connections. Namely I have a tracking unit connected up via serial so I can see when it makes a connection to my server, but then a connection is never made on the server side. On the serial it show "Connected" and is sending the data but nothing at the server – user3714510 Aug 19 '15 at 12:33
0

I managed to solve the problem I have been having with my server and Tracking Units. What has happened, we are using a custom APN network from our local service provider, and we are experiencing problem from their side. Connections will get lost and when a connection is made with my server, the APN will just drop the connection. That is why I was getting a lot of SYN_RECV connections that only quits after timeouts.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.