9

We have recently implemented HAProxy for stackoverflow.com. We decided on using TProxy to maintain the source address for clients connecting so our logs and other IIS modules which depend on the client IP address would not require modification. So the packets arrive spoofed as if they have come from an external internet IP address, when in reality they came from a local 192.168.x.x HAProxy IP on our local network.

Both of our web servers have two NICs - one routable class B address on the public internet with a static IP, DNS, and default gateway and one private unroutable class C address configured with a default gateway pointed at the private IP for HAProxy. HAProxy has two interfaces - one public and one private and performs the job of routing packets transparently between interfaces and directing traffic to the appropriate web server.

Ethernet adapter Internet:

   Description . . . . . . . . . . . : network card #1
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 69.59.196.217 (Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.240
   Default Gateway . . . . . . . . . : 69.59.196.209
   DNS Servers . . . . . . . . . . . : 208.67.222.222
                                       208.67.220.220
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Private Local:

   Description . . . . . . . . . . . : network card #2
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.0.2 (Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.0.50
   NetBIOS over Tcpip. . . . . . . . : Enabled

We have disabled automatic metrics on each of the web servers and assigned the routable public class B a metric of 10 and our private interface a metric of 20.

We have also set both of these registry keys:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"DeadGWDetectDefault"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"EnableDeadGWDetect"=dword:00000000

About twice per day we see issues where one of the web servers cannot contact DNS or make connections out to any other servers on the public internet.

We suspect dead gateway detection is falsely detecting an outage on the public gateway and is switching all traffic to the private gateway which has no DNS access at this point but have no way of verifying this.

  1. Is there a way to know if dead gateway detection is running or even an option in Windows 2008 server?

  2. If so, is there a way to disable dead gateway detection in Windows 2008 server?

  3. If not could there be other reasons that we lose the ability to resolve DNS or connect out for a short time?

Improve this question
5
  • 1
    While this setup is sometimes frowned upon (see blogs.technet.com/timmcmic/archive/2009/04/26/… ), it works awesomely for us - all traffic coming from HAProxy to our IIS sites looks like it's still coming from the original IP address. This saves untold amounts of time, as we'd have to (find out how to) configure IIS and its myriad plug-ins to use an HTTP_X_FORWARDED_FOR header.
    – Jarrod Dixon
    Sep 11, 2009 at 10:05
  • 1
    Why do you have a gateway configured on the 192.168.0.2 interface? You can configure an empty default gateway (and in fact this is what Windows prompts you to do when you have two interfaces).
    – Portman
    Sep 13, 2009 at 10:51
  • @Portman - because our web boxes are seeing the traffic with the originating client IPs intact, the responses aren't going to be sent to our network - that's why we have to have a default gateway to our HAProxy box.
    – Jarrod Dixon
    Sep 13, 2009 at 19:10
  • @Jarrod - that configuration seem suspicious. What about if you want to run a non-balanced website on that web server? The response will be routed through HAProxy? How would you handle something like remote desktop? I realize this does not address the question, but this does seem like a case of You're Doing It Wrong, which is what daivdsmalley is (politely) saying.
    – Portman
    Sep 13, 2009 at 20:34
  • 4
    @Jeff/Geoff/Jarrod - I hate to state the obvious, but you guys are software devs, why not hire in someone who is a specialist for a day to fix? It's all very nice to get your hands dirty but there's a clear knowledge gap here, it's intermittently affecting the business and you've clearly spent a fair bit of valuable time not utilising your core skills which is development. Trust me, get someone in to fix and then pick his/her brains after you've got it working. Hell, even as webhosters we need to get folks in to bridge these gaps when it's mission critical/service affecting.
    – Kev
    Sep 14, 2009 at 3:15

4 Answers 4

Reset to default
5

Those Dead Gateway Detection DWORDs are useless on Windows Server 2008. The only reason they exist is for compatibility reasons. The TCP/IP driver and Windows router components don't look for these values anymore.

I suspect this feature was rolled into Auto-Tuning, which debuted in Windows Vista. Try executing the following in an elevated command prompt (and reboot):

netsh int tcp set global autotuninglevel=disabled


Update (added September 13, 2009 @7:58PM EST)

If that doesn't work, we'll need more diagnostic output. Start a (circular) trace with either the NetConnection or LAN scenarios and let it continue running until the problem occurs.

netsh trace start scenario=NetConnection maxSize=512

(Example: Starts the NetConnection tracing scenario, with a maximum trace log size of 512MB)

You can open the resulting trace in Network Monitor 3.3, just make sure you install the latest parsers.

Improve this answer
2
  • good idea, but didn't seem to work either.. just experienced a 5 minute outgoing traffic outage -- which mysteriously fixed itself.
    – Jeff Atwood
    Sep 13, 2009 at 22:24
  • @Jeff: Hmm, we need more data Captain! See edit above.
    – Rafael Rivera
    Sep 14, 2009 at 0:00
5

We were not able to arrive at a conclusive result as to why we could not control the behavior of Dead Gateway Detection.

Rather than spend a ton of time troubleshooting this issue we opted to make our HAProxy instance route traffic to the gateway outbound and set both web servers default gateway to the IP of haproxy and removed the internal gateway address. 

  [ soweb1 ] 69.59.196.220, GW=69.59.196.211 [haproxy]
       |
       +---- [haproxy] 69.59.196.211, GW 69.59.196.209
       |
    [ gw ] 69.59.196.209

Now there is only one default gateway which eliminates our issue because dead default gateway detection is no longer used.

Improve this answer
4

I would question why you even need to change the default gateway to be HAproxy at all. Generally you should not change your default gateway at all unless you're pointing it at a highly available N+1 setup where the gateway IP can failover to another router/machine in the event of something bad happening. If something happened to your HAproxy machine and you didn't have any out-of-band access, then the web servers would just drop off the internet.

As I believe the reason you may be doing this is because you are using Tproxy in your setup to make the clients IP address appear in your logs and not the proxy server's IP, could I suggest that you do this instead

  1. Add "option forwardfor ..." to your HAproxy config
  2. Install the x-forwarded-for ISAPI filter
  3. Remove tproxy from your setup
  4. Change the default gateway back to the same gateway you were using before with direct connection the internet

I don't have a Windows machine to test this on but I believe it should result in the desired effect without the undesired loss of connectivity.

Improve this answer
4
  • I only just spotted your comment on the original question regarding this setup. However, I would doubt "it works awesomely for us" if your servers are loosing internet connectivity :)
    – davidsmalley
    Sep 13, 2009 at 8:49
  • 3
    Alternatively, you could look at a much more robust solution such as ldirectord+heartbeat which just redirects traffic at a kernel level, as such there is no proxying involved at all. I use this setup extensively and it works great. linuxvirtualserver.org/docs/ha/heartbeat_ldirectord.html
    – davidsmalley
    Sep 13, 2009 at 8:57
  • We've looked at using that x-forwarded-for header and IIS filters to alter the logs, but we don't know how (or if) our other optional IIS modules also use the header in their operation.
    – Jarrod Dixon
    Sep 13, 2009 at 19:14
  • Thanks for that linuxvirtualserver.org/HighAvailability.html link - the information there is amazing! I'm beyond ignorant on these subjects (which is why I'm not the one setting all this up!), but I'm trying to learn as fast as possible. Perhaps we can use heartbeat+ldirectord similar to how linuxvirtualserver.org/docs/ha/ultramonkey.html does it with our favorite HAProxy.
    – Jarrod Dixon
    Sep 13, 2009 at 19:18
-1

When internet access is involved (usually) then default gateways should only EVER be used to denote a path to the INTERNET. If you have multiple default gateways defined, the OS router can't decide which one to use, and if one default gateway points down a cul-de-sac (e.g. your multi-segment LAN) then the packets forwarded there for the internet are not going to make it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.