12

Our Windows servers are registering IPv6 AAAA records with our Windows DNS servers. However, we don't have IPv6 routing enabled on our network, so this frequently causes stall behaviours.

Microsoft RDP is the worst offender. When connecting to a server that has a AAAA record in DNS, the remote desktop client will try IPv6 first, and won't fall back to IPv4 until the connection times out. Power users can work around this by connecting to the IP address directly. Resolving the IPv4 address with ping -4 hostname.foo always works instantly.

What can I do to avoid this delay?

  • Disable IPv6 on client?
  • Disable IPv6 on the server?
  • Mask IPv6 records on the user-facnig DNS recursor?
  • Prevent registration of IPv6 AAAA records on the Microsoft DNS server?
    • I don't think that's even possible.

At this point, I'm considering writing a script that purges all AAAA records from our DNS zones. Please, help me find a better way.


UPDATE: DNS resolution is not the problem. As @joeqwerty points out in his answer, the DNS records are returned instantly. Both A and AAAA records are immediately available. The problem is that some clients (mstsc.exe) will preferentially attempt a connection over IPv6, and take a while to fall back to IPv4.

This seems like a routing problem. The ping command produces a "General failure" error message because the destination address is unroutable.

C:\Windows\system32>ping myhost.mydomain
Pinging myhost.mydomain [2002:1234:1234::1234:1234] with 32 bytes of data:
General failure.
General failure.
General failure.
General failure.
Ping statistics for 2002:1234:1234::1234:1234:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

I can't get a packet capture of this behaviour. Running this (failing) ping command does not produce any packets in Microsoft Network Monitor. Similarly, attempting a connection with mstsc.exe to a host with an AAAA record produces no traffic until it does a fallback to IPv4.

UPDATE: Our hosts are all using publicly-routable IPv4 addresses. I think this problem might come down to a broken 6to4 configuration. 6to4 behaves differently on hosts with public IP addresses vs RFC1918 addresses.

UPDATE: There is definitely something fishy with 6to4 on my network. When I disable 6to4 on the Windows client, connections resolve instantly.

netsh int ipv6 6to4 set state disabled

But as @joeqwerty says, this only masks the problem. I'm still trying to find out why IPv6 communication on our network is completely non-working.

11
  • 11
    Finish deploying IPv6 on your network, of course. Oct 25, 2013 at 22:16
  • 1
    Have you run a network capture on a client to confirm this IPv6 resolution failure/delay?
    – joeqwerty
    Oct 25, 2013 at 22:28
  • 1
    As an aside, Microsoft provides several convenient Fixit tools to disable/enable various IPv6 components: support.microsoft.com/kb/929852
    – joeqwerty
    Oct 25, 2013 at 22:36
  • 1
    @joeqwerty The servers and users are on separate subnets, but everything is one big site. We aren't using split-brain DNS, so there is no concept of "internal DNS".
    – Nic
    Oct 25, 2013 at 22:43
  • 2
    Also I think you will find this article from RIPE as well as RFC 6343 very interesting and relevant reading. My personal recommendation to you would be to dump 6to4 entirely. Oct 26, 2013 at 18:48

4 Answers 4

10

This question is pretty interesting and I must admit that I've never seen this behavior. In doing some fiddling around to try and understand it better, I took a snippet of nslookup querying for one of my W2K8R2 RDS servers from another W2K8R2 server and I also captured a snippet of an RDP session to the same RDS server from the same test server. Nslookup showed no delay in returning the IPv6 record and the nslookup showed my test server querying for the IPv4 record before querying for the IPv6 record. The time delta in the capture shows no appreciable delay (that I can ascertain) in either query.


enter image description here


enter image description here


EDIT

Now you're on to something.

Make sure you're capturing traffic for the Microsoft 6To4 adapter, otherwise you won't see IPv6:

enter image description here


Here's the nslookup result for my RDS server. Make note of the IPv6 addresses:

enter image description here


Now here's a snippet of my capture:

enter image description here


And finally, here's a snippet from netstat showing the connection:

enter image description here


So clearly, as you've confirmed, DNS resolution isn't the problem. The problem is that the RDP connection prefers IPv6 over IPv4 (which is the default for Windows - Windows prefers IPv6 over IPv4) and because IPv6 isn't functioning properly it's causing the delay (as you've stated) when falling back from IPv6 to IPv4. You could fix this by configuring the clients to prefer IPv4 over IPv6, but I think that would merely be masking the problem. The better solution would be to figure out why IPv6 isn't working and fix that. I don't know enough about IPv6 to help but my guess is that the IPv6 records being returned by DNS are "local" addresses valid only on the subnet where the RDS hosts exist and since the clients are in a different subnet, they can't reach those IPv6 addresses.

4
  • Bro, do you even RFC 1918?
    – Ryan Ries
    Oct 26, 2013 at 0:20
  • LOL. They got in early, were allocated their own /24 block and chose to use it internally instead of dealing with NAT. They use about 80% of the block and have taken the stance "if it's not broke don't fix it".
    – joeqwerty
    Oct 26, 2013 at 0:35
  • @joeqwerty I updated my question with some clarifications. nslookup works fine but ping fails with General Failure.
    – Nic
    Oct 26, 2013 at 17:20
  • @joeqwerty Thanks for all of your input. I've posted an answer to this question which clarifies what happened in my environment, and suggests what other people might do in a similar situation.
    – Nic
    Oct 28, 2013 at 0:47
9

The IPv6 transition technology called 6to4 is infamous for causing problems like this one. There are several factors at work. Individually they are harmless, but the combined effect is that end users can experience connection delays.

A list of enabling factors and thoughts on their mitigation is presented below.


Windows enables 6to4 by default

If your hosts are running a recent version of Windows (Vista or later), Windows will opportunistically enable 6to4 tunnelling when a publicly routable IPv4 address is available. Critically, this applies to both servers and clients.

To find out whether a system is using 6to4, run ipconfig and look for an IPv6 address that starts with the 6to4 prefix 2002:. It would look something like this.

C:\> ipconfig
Tunnel adapter 6TO4 Adapter:
IPv6 Address. . . . . . . . . . . : 2002:1111:2222::1111:2222
  • If your endpoints are connected to Active Directory, you can use Group Policy to disable transition protocols like 6to4 and Teredo. This is well-documented in KB929852. (Applying this to either your clients or servers would be sufficient, but if you're taking this step it probably makes sense to disable it everywhere, on both clients and servers.)
  • If you only manage a few hosts, you can disable 6to4 on a case by case basis. This is much better than disabling IPv6 entirely. netsh int ipv6 6to4 set state disabled
  • Use a different client operating system. For example, Mac OS X does not have 6to4 enabled by default.

Publicly routable IPv4 addresses are being used

6to4 only works on hosts that have publicly routable IPv4 addresses so this problem never affects hosts behind a NAT firewall.

  • You can move clients and/or servers behind a NAT firewall and start using RFC1918 addressing. But in some cases, publicly routable addresses are actually preferred. Changing the addressing of an entire network might also be an unrealistic choice.

6to4 is not functioning correctly on the network

It is infamously difficult to troubleshoot 6to4 in anycast mode. It is so troublesome that there was a formal request to the IETF that 6to4 should be reclassified as historic. In the opinion of this author, 6to4 has been deprecated.

In brief, 6to4 works by encapsulating IPv6 packets into IPv4 packets using a protocol called 6in4 (IP protocol=41). The IPv4 packets are addressed to anycast address 192.88.99.1 in hopes that it will arrive at a working 6to4 relay somewhere on the internet. It might even be geographically nearby, if you're lucky.

In practice, some 6to4 relays are set up incorrectly, and a lot of networks don't even allow 6in4 traffic to cross the firewall. Typically this happens when a firewall allows all outbound traffic, but doesn’t explicitly allow IP protocol 41 packets to return through the firewall. (TODO note the relevant RFC for troubleshooting.) This failure ("inbound black hole") and many others are described in RFC 6343.

  • Set up your firewall to loudly reject IP protocol 41 (with TCP resets) when sent from hosts internal to your network. This should result in a "fail fast" behaviour that makes more sense than non-deterministic connection delays. This has been demonstrated to work in limited test environments.
  • Ask your ISP or first-hop transit provider to set up a working 6to4 relay. If this is done right, it will result in the best experience for end users. Any end user with a publicly routable IPv4 address would be able to participate in the IPv6 internet.

Dynamic DNS registration

In a typical Active Directory environment, every computer is permitted to register its own addresses with the DNS server. When a host is multihomed, it will register all of its addresses, even from a 6to4 tunnel.

Most internet services don't use dynamic DNS, so this problem is typically restricted to enterprise sites where the clients and servers are all "internal" to the same network.

  • You may choose to disable dynamic DNS updates. Then if you don't put any AAAA resource records into the zone file, they will never be served. However, dynamic DNS is often desirable for internal DNS servers. (If you do this, make sure to also delete any AAAA records that might already be present.)
  • Set up the DNS server to not provide answers for AAAA resource records. But don't do this, because it will really give you trouble when you want to start implementing IPv6. (Does anybody know of a free/open source DNS firewall?)

Client application does not fail gracefully

Microsoft's RDP client is one example of a client application that does not gracefully deal with IPv6 routing problems. Most web browsers are better at dealing with IPv6 edge cases like this one, so they don’t tend to show this behaviour.

  • Try using a different client. Maybe you'll get lucky.
2
  • I'd extend the discussion about 6to4 issues with a mention of how RFC 6598 addresses could make the problem worse. Most software which automatically enables 6to4 if a public IPv4 address is available was written before that RFC. Hence an RFC 6598 address will naturally be detected as if it was a public IPv4 address. But it isn't, and running 6to4 on a RFC 6598 address is not going to work. If you see any IPv6 address from the 2002:6440::/26, then you have run into the 6to4 + RFC 6598 problem.
    – kasperd
    Aug 2, 2014 at 13:40
  • The 6to4 anycast relay is only relavent when communicating between a 6to4 host and a native IPv6 host, it's not relavent when communicating between two 6to4 hosts (ironically this means that adding native IPv6 to some but not all hosts could make things worse). Sep 19, 2016 at 16:48
2

I realize it's not very helpful for this situation, but for implementors facing a similar dilemma, there is an implementation technique known as "Happy Eyeballs" (RFC 6555) that specifies a technique for connecting to ipv4 and ipv6 simultaneously and choosing whichever connects first.

0

Here was my solution. By default Windows gives IPv6 routes a higher priority than IPv4 routes. If you edit the IPv6 prefix policy, you can change this behavior to make it use IPv4 in preference to IPv6.

To make sure all the systems in my network are set up the same way, I put the following commands into a .bat script run during software installation after building or refurbishing a machine.

netsh int ipv6 isatap set state disabled
netsh int ipv6 6to4 set state disabled
netsh interface teredo set state disable

netsh interface ipv6 delete prefixpolicy ::1/128
netsh interface ipv6 delete prefixpolicy ::/0
netsh interface ipv6 delete prefixpolicy 2002::/16
netsh interface ipv6 delete prefixpolicy ::/96
netsh interface ipv6 delete prefixpolicy ::ffff:0:0/96
netsh interface ipv6 delete prefixpolicy 2001::/32

netsh interface ipv6 add prefixpolicy ::1/128 50 0
netsh interface ipv6 add prefixpolicy ::ffff:0:0/96 40 1
netsh interface ipv6 add prefixpolicy ::/0 30 2
netsh interface ipv6 add prefixpolicy 2002::/16 20 3
netsh interface ipv6 add prefixpolicy ::/96 10 4
netsh interface ipv6 add prefixpolicy 2001::/32 5 5

To explain what this does:

The first 3 lines disable the builtin tunneling interfaces, as they are redundant for most networks. You might want to not use those 3 lines if you are not giving your machines IPv6 addresses of their own, in my case I have a DHCPv6 server and associated infrastructure assigning IPv6 for tunneled connectivity

The second block of commands deletes all existing IPv6 routing prefix policies.

The third block then recreates the IPv6 prefix policies, but uses a different set of priorities. Like so the prefix corresponding to IPv4 is given preference over IPv6, and the machine will then want to use IPv4 unless the application specifies the use of IPv6.

This solution retains functional dual-stack capability, but the preference to use IPv4 means that sites with incomplete, unreliable, or poor performing IPv6 will avoid using it unless told to by a program on the system.

It is my opinion that making operating systems use IPv6 in preference to IPv4 is actually hindering adoption. During the transitional period, there will be times where a host thinks it has IPv6 connectivity but does not actually have a fully functional connection, leading to software malfunctions and large delays. Many people I know have disabled IPv6 entirely at their router as a workaround for ISPs deploying IPv6 in a broken fashion initially before establishing full connectivity, and these people will simply forget to enable it again leaving them without IPv6 until they reconfigure their router again.

1
  • +1 for disabling tunnelling, -1 for preferring IPv4. This is not an issue for most people, and should only be applied for specific users in specific circumstances. Jul 10, 2015 at 16:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .