2

We suffer from a connection refused problem when the users of our web site try to open it. This problem happens in a random manner, about once or twice a month, and problem continues for a few hours. Also when happening, almost all connections are rejected by connection refused error. but there are successful connections meanwhile.

  • OS: Win 2012 R2 Standard hosted on ESXI 6
  • IIS 8.5
  • Web server is hosting an ASP.NET application.
  • Windows Firewall is on.
  • Average current connection on server: ~3500 (based on Web Service\Current connection performance monitor counter)
  • Total RAM: 40GB
  • CPU: 30 cores, 2.30 GHz

There is plenty of RAM (more than ~60%) and CPU (more than ~70%) available while this problem happens. Also we checked the network firewall and apparently traffic is passing through network firewall without problem and problem happens at the server level. And we can not even open the web site by doing Remote desktop and trying to open in locally.

We checked about exhausted port problem and it seems that is not the problem. The number of SYN packets are high, but its similar to other days when everything is fine.

This is one day summery of HTTPERR log:

s-reason    COUNT(ALL *)
Timer_ConnectionIdle    462040
Timer_MinBytesPerSecond 27555
Request_Cancelled   1757
Timer_EntityBody    428
Forbidden   247
URL 130
Hostname    117
BadRequest  102
Connection_Dropped  96
Client_Reset    88
Connection_Dropped_List_Full    40
Verb    10
Header  7
Connection_Abandoned_By_ReqQueue    1

Any help is really appreciated to find the reason why we get connection refused when trying to open web site hosted on this server.

Improve this question

1 Answer 1

Reset to default
0

Are you running in a virtual environment, or a physical machine? (Edit, just re-read, and saw ESXI 6. So Virtual it is then.)

You've got a VMware VM, are you using the standard install NIC, or are you using the VM providers specific NIC? (ie: Intel vs. VMWare)

We have a similar issue, but is much less persistent when it occurs. (Ours is exposed when an automated script runs to check 30 sites up/down status, but only effects a half dozen LWP gets.)

(Sorry, can't use comments yet, don't have the rep for it)

Edit 2

As per this TechNet Link Mohammad found, SYN Attack Protect is on by default in >= Vista. Which is what I found yesterday, but unlike what I read yesterday, the RegEdits in Edit 1 apparently don't make it more aggressive or active.

I've taken a temporary approach of blocking IPs at the firewall to see what happens. Excess SYN_RECEIVED connections drop, and then eventually rise again on another IP (As you would expect).

Edit 1 - [Possibly debunked?]

If you haven't got to reading all the comments below, it looks like this is headed in the direction of a SYN Attack (for both of us).

I'm currently trialling the following changes in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ on a development server for testing:

SynAttackProtect: https://technet.microsoft.com/en-us/library/cc938202.aspx TcpMaxConnectResponseRetransmissions: https://technet.microsoft.com/en-us/library/cc938208.aspx TcpTimedWaitDelay: https://technet.microsoft.com/en-us/library/cc938217.aspx TcpMaxHalfOpenRetried: https://technet.microsoft.com/en-us/library/cc938213.aspx TcpMaxPortsExhausted: https://technet.microsoft.com/en-us/library/cc938214.aspx TcpTimedWaitDelay: https://technet.microsoft.com/en-us/library/cc938217.aspx

Somewhat like what is detailed here: https://alnitech.com/news/how-to-protect-your-windows-server-from-syn-flood/

Useful note - some TCP / UDP Port usages. Especially if you're considering increasing ports in the ephemeral range. (ie: netsh int ipv4 set dynamicportrange tcp start=45536 num=20000) https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers

Improve this answer
17
  • We use Intel NIC provider. Do you mean you face "Connection refused" when you run a perl script which uses LWP to get IIS site status?
    – Mohammad Reza Sadreddini
    Sep 4, 2017 at 4:20
  • Yes. It has become much more prevalent lately. And no sooner had I written the above, I got a list of about 20 Connection refused from LWP. I Can only assume it's stemming from the same thing, as all the wisdom I've read says that the feedback Perl's giving is straight from the destination server. Although, the IIS logs and the FailedHTTPReq logs don't show anything of note. (As an aside, I've set up Verbose FailedRequestLogging to see if this will shed any light on this.)
    – Mark
    Sep 4, 2017 at 4:32
  • Getting back to the point, there is a VMWare version of the NIC, e1000 I think? ... although, it might be the VMXNET3 NIC. You'd need to seek information from your host provider from here. I've reached the edge of my knowledge. (Note: Split comments in to two for ease of readability)
    – Mark
    Sep 4, 2017 at 4:35
  • I think I remember now, the e1000 IS the Intel one, And the VMXNET3 is the one you want (the VMWare One)
    – Mark
    Sep 4, 2017 at 5:14
  • We have this problem on all our servers, I check one of them is using VMXNET3 and problem still exists. We are struggling with problem about 2 years... Next time this problem happens, we are going to use Microsoft Network Monitor to trace packet in order to find a clue.
    – Mohammad Reza Sadreddini
    Sep 4, 2017 at 9:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .