Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have recently implemented HAProxy for stackoverflow.com. We decided on using TProxy to maintain the source address for clients connecting so our logs and other IIS modules which depend on the client IP address would not require modification. So the packets arrive spoofed as if they have come from an external internet IP address, when in reality they came from a local 192.168.x.x HAProxy IP on our local network.

Both of our web servers have two NICs - one routable class B address on the public internet with a static IP, DNS, and default gateway and one private unroutable class C address configured with a default gateway pointed at the private IP for HAProxy. HAProxy has two interfaces - one public and one private and performs the job of routing packets transparently between interfaces and directing traffic to the appropriate web server.

Ethernet adapter Internet:

   Description . . . . . . . . . . . : network card #1
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 69.59.196.217 (Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.240
   Default Gateway . . . . . . . . . : 69.59.196.209
   DNS Servers . . . . . . . . . . . : 208.67.222.222
                                       208.67.220.220
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Private Local:

   Description . . . . . . . . . . . : network card #2
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.0.2 (Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.0.50
   NetBIOS over Tcpip. . . . . . . . : Enabled

We have disabled automatic metrics on each of the web servers and assigned the routable public class B a metric of 10 and our private interface a metric of 20.

We have also set both of these registry keys:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"DeadGWDetectDefault"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"EnableDeadGWDetect"=dword:00000000

About twice per day we see issues where one of the web servers cannot contact DNS or make connections out to any other servers on the public internet.

We suspect dead gateway detection is falsely detecting an outage on the public gateway and is switching all traffic to the private gateway which has no DNS access at this point but have no way of verifying this.

  1. Is there a way to know if dead gateway detection is running or even an option in Windows 2008 server?

  2. If so, is there a way to disable dead gateway detection in Windows 2008 server?

  3. If not could there be other reasons that we lose the ability to resolve DNS or connect out for a short time?

share|improve this question
1  
While this setup is sometimes frowned upon (see blogs.technet.com/timmcmic/archive/2009/04/26/… ), it works awesomely for us - all traffic coming from HAProxy to our IIS sites looks like it's still coming from the original IP address. This saves untold amounts of time, as we'd have to (find out how to) configure IIS and its myriad plug-ins to use an HTTP_X_FORWARDED_FOR header. –  Jarrod Dixon Sep 11 '09 at 10:05
1  
Why do you have a gateway configured on the 192.168.0.2 interface? You can configure an empty default gateway (and in fact this is what Windows prompts you to do when you have two interfaces). –  Portman Sep 13 '09 at 10:51
    
@Portman - because our web boxes are seeing the traffic with the originating client IPs intact, the responses aren't going to be sent to our network - that's why we have to have a default gateway to our HAProxy box. –  Jarrod Dixon Sep 13 '09 at 19:10
    
@Jarrod - that configuration seem suspicious. What about if you want to run a non-balanced website on that web server? The response will be routed through HAProxy? How would you handle something like remote desktop? I realize this does not address the question, but this does seem like a case of You're Doing It Wrong, which is what daivdsmalley is (politely) saying. –  Portman Sep 13 '09 at 20:34
4  
@Jeff/Geoff/Jarrod - I hate to state the obvious, but you guys are software devs, why not hire in someone who is a specialist for a day to fix? It's all very nice to get your hands dirty but there's a clear knowledge gap here, it's intermittently affecting the business and you've clearly spent a fair bit of valuable time not utilising your core skills which is development. Trust me, get someone in to fix and then pick his/her brains after you've got it working. Hell, even as webhosters we need to get folks in to bridge these gaps when it's mission critical/service affecting. –  Kev Sep 14 '09 at 3:15
add comment

3 Answers 3

Those Dead Gateway Detection DWORDs are useless on Windows Server 2008. The only reason they exist is for compatibility reasons. The TCP/IP driver and Windows router components don't look for these values anymore.

I suspect this feature was rolled into Auto-Tuning, which debuted in Windows Vista. Try executing the following in an elevated command prompt (and reboot):

netsh int tcp set global autotuninglevel=disabled


Update (added September 13, 2009 @7:58PM EST)

If that doesn't work, we'll need more diagnostic output. Start a (circular) trace with either the NetConnection or LAN scenarios and let it continue running until the problem occurs.

netsh trace start scenario=NetConnection maxSize=512

(Example: Starts the NetConnection tracing scenario, with a maximum trace log size of 512MB)

You can open the resulting trace in Network Monitor 3.3, just make sure you install the latest parsers.

share|improve this answer
    
good idea, but didn't seem to work either.. just experienced a 5 minute outgoing traffic outage -- which mysteriously fixed itself. –  Jeff Atwood Sep 13 '09 at 22:24
    
@Jeff: Hmm, we need more data Captain! See edit above. –  Rafael Rivera Sep 14 '09 at 0:00
add comment
up vote 5 down vote accepted

We were not able to arrive at a conclusive result as to why we could not control the behavior of Dead Gateway Detection.

Rather than spend a ton of time troubleshooting this issue we opted to make our HAProxy instance route traffic to the gateway outbound and set both web servers default gateway to the IP of haproxy and removed the internal gateway address.

  [ soweb1 ] 69.59.196.220, GW=69.59.196.211 [haproxy]
       |
       +---- [haproxy] 69.59.196.211, GW 69.59.196.209
       |
    [ gw ] 69.59.196.209

Now there is only one default gateway which eliminates our issue because dead default gateway detection is no longer used.

share|improve this answer
add comment

I would question why you even need to change the default gateway to be HAproxy at all. Generally you should not change your default gateway at all unless you're pointing it at a highly available N+1 setup where the gateway IP can failover to another router/machine in the event of something bad happening. If something happened to your HAproxy machine and you didn't have any out-of-band access, then the web servers would just drop off the internet.

As I believe the reason you may be doing this is because you are using Tproxy in your setup to make the clients IP address appear in your logs and not the proxy server's IP, could I suggest that you do this instead

  1. Add "option forwardfor ..." to your HAproxy config
  2. Install the x-forwarded-for ISAPI filter
  3. Remove tproxy from your setup
  4. Change the default gateway back to the same gateway you were using before with direct connection the internet

I don't have a Windows machine to test this on but I believe it should result in the desired effect without the undesired loss of connectivity.

share|improve this answer
    
I only just spotted your comment on the original question regarding this setup. However, I would doubt "it works awesomely for us" if your servers are loosing internet connectivity :) –  davidsmalley Sep 13 '09 at 8:49
3  
Alternatively, you could look at a much more robust solution such as ldirectord+heartbeat which just redirects traffic at a kernel level, as such there is no proxying involved at all. I use this setup extensively and it works great. linuxvirtualserver.org/docs/ha/heartbeat_ldirectord.html –  davidsmalley Sep 13 '09 at 8:57
    
We've looked at using that x-forwarded-for header and IIS filters to alter the logs, but we don't know how (or if) our other optional IIS modules also use the header in their operation. –  Jarrod Dixon Sep 13 '09 at 19:14
    
Thanks for that linuxvirtualserver.org/HighAvailability.html link - the information there is amazing! I'm beyond ignorant on these subjects (which is why I'm not the one setting all this up!), but I'm trying to learn as fast as possible. Perhaps we can use heartbeat+ldirectord similar to how linuxvirtualserver.org/docs/ha/ultramonkey.html does it with our favorite HAProxy. –  Jarrod Dixon Sep 13 '09 at 19:18
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.