We have some weird problems, when external servers try to connect to our Website on Port 443(TCP) which is an Apache2+Passenger and a Rails Application.
The problem only happens rarely. I started to investigate that by enabling iptables logging using
iptables -I INPUT -m state --state NEW,INVALID -j LOG --log-prefix "iptables [INPUT]: " (did this also for FORWARD (normally not used) and OUTPUT as well).
Now every new connection is logged into syslog. Now ive seen normal Connections on that syslog and afterwars the connection to the Vhost (normal working request) but sometimes (in about 1 of 100 connections) i only see the new connection in the iptables log but the connection seems to not reach the apache2 (or the passenger instances) (No apache Log entry and no result from the connection). When doing this with curl, we get a timeout or an empty string back. We just sent over a debug-enabled curl request code snippet for enhanced logging what the curl error looks like.
We`re using Debian Squeeze (stable) running in a VMWare ESXi environment with bridged ethernet with E1000 NICs (virtualized). But as the packets reaching the VM-OS itself i dont think that anything of that causes the issue.
I also checked ip_conntrack which has about 40 to 50 connections and Squeezes Limit is about 65k, so thats not the problem. Also we dont have any Dropped Packages on the NIC`s itself.
Kernel-Version: Linux servername 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64 GNU/Linux Apache2: Server version: Apache/2.2.16 (Debian) - Server built: Nov 30 2012 08:58:38 Passenger: Phusion Passenger version 3.0.12 Ruby: RVM 1.17.9 + Ruby 1.9.3-p362
Chain INPUT (policy ACCEPT) target prot opt source destination
LOG all -- anywhere anywhere state INVALID,NEW LOG level warning prefix `iptables [INPUT]: '
Chain FORWARD (policy ACCEPT) target prot opt source destination
LOG all -- anywhere anywhere state INVALID,NEW LOG level warning prefix `iptables [FORWARD]: '
Chain OUTPUT (policy ACCEPT) target prot opt source destination
LOG all -- anywhere anywhere state INVALID,NEW LOG level warning prefix
iptables [OUTPUT]: '