I am troubleshooting network problems for a client, remotely. The problem they have is that they get "connection timed out" occasionally between a web server and and a back-end search server. They can reproduce this behavior easily using a staging server. I have asked them run Wireshark on both servers and what I find is that SYN packets are sent, over and over again. And often they are not visible on the receiving end. I am wondering what you guys think the reason could be?
My first thought was the firewall that is between the servers. Now they say they connected the back-end search server to the same network as the web server, which makes me puzzled.
More details: I assume the servers are Windows Server 2008. I have never been to the client's location. The web server is using WCF with Transport Security turned on to access the back-end servers. They seem to be able to rule out heavy load as these problems appear also at light load.
For me it sounded obvious that it has to be something in the network that causes the SYNs to not appear at the destination but now they say they have turned off firewall rules, turned off Windows firewalls and even put the servers on the same network. And I'm clueless.
Update: The latest test they've made is to run a console app (simulating repeated web requests) on a server on the same subnet as the search server. And both servers run as VMWare instances.