I have a Windows Server 2012 R2 IIS web server, running on VMWare esxi 5.5, hosting multiple websites with different IP addresses. This server recently has been refusing connections randomly across all websites. When the server stops accepting connections, there are tons of SYN_RECEIVED entries in netstat. Sometimes from the same IP address across all the hosted IP addresses other times from different IP addresses but a lot more of them. These entries sit for a while and then will disappear (assuming a timeout period is reached).
I've read this is probably a SYN flood attack, but that Windows is supposed to have a built in way of handling these attacks. I looked at this article, https://blogs.technet.microsoft.com/nettracer/2010/06/01/syn-attack-protection-on-windows-vista-windows-2008-windows-7-windows-2008-r2-windows-88-1-windows-2012-and-windows-2012-r2/, and did a netsh trace, but there's nothing in the logs that indicate SYN attack protection has been initiated. This makes me wonder whether SYN attack protection is working. The web server does have the built in Windows firewall disabled. Could that cause the SYN attack protection to be disabled?
I also read that increasing the server resources could help. I doubled the CPU and RAM for the virtual machine, but this problem still crops up every week. Usually for a day and then goes back to normal. When the server stops responding to connections, netstat will be filled with a bunch of SYN_RECEIVED entries and it will take several minutes to complete. At this time the CPU usage is really low, around 5-10%, and the RAM usage is half of what is available. After the suspected attack, netstat will return all the results in a few seconds.
How can I stop this connection refused thing from happening? Any tips or tricks? Does the disabled windows firewall have something to do with the mitigating the attacks.