2

I just spent about 6 hours trying to figure this out, and I now believe CentOS/Linux is unable to bind to a specific virtual IPv4 address when connecting to a hostname that has an IPv6 address. This is a problem on servers that have multiple IP addresses.

I am using Centos 6 (Linux kernel 2.6.32-573.12.1.el6.x86_64)

To reproduce this big:

  1. Find a Linux machine with at least a /29 IPv4 public address space, and an IPv6 public IP.
  2. Alias at least one additional IPv4 to the main interface (eth0 or otherwise). For this example, I will say 30.0.0.1 as the machine's primary eth0 IPv4 address, and 30.0.0.2 is an alias bound to eth0:2 on a network of 30.0.0.0/29.
  3. Find a hostname that has both IPv4 and IPv6 addresses. For example, www.microsoft.com.
  4. telnet -b 30.0.0.2 www.microsoft.com 80 (This tests making an outbound connection using a specific ipv4 address)
  5. The IPv4 request connects successfully after unsuccessfully trying the hostname's IPv6 addresses, but the TCP connection actually initiates from the machine's main IP (30.0.0.1), not the IP you want (30.0.0.2). Netstat says otherwise, but it is WRONG. If you connect to a hostname that you own and can view its logs, the connection comes from the main IP (30.0.0.1) not the IP you want (30.0.0.2).
  6. Try another domain that does not have an IPv6 address, like: telnet -b 30.0.0.2 serverfault.com. It works. It makes the connection from the IP you want.

This is a problem, because certain programs, like mail (exim), need to use certain IPs when making outbound TCP requests, which is not necessarily the main machine's IP. Certain clients/programs on the machine rely on ACLs or reverse DNS to match up correctly when making outbound TCP connections.

So if anyone else is noticing the same weird problem where their program cannot bind to the correct interface when making outgoing connections, this is probably why.

This problem only effects IPv4 connections. IPv6 connections correctly bind to whatever outgoing IP you have on the machine.

This is not an issue with telnet. I tested this problem using my mail server (exim) too, and got similar results. It makes IPv4 connections from the wrong IP if the target hostname has an IPv6 address.

Maybe someone has a solution to this strange problem, but at this time, I think it may be a Linux networking bug.

Ps- If someone wonders why not just make an IPv6 connection if the hostname resolves to an IPv6 address...sometimes the IPv6 address is down or the connection cannot establish, then it reverts to its IPv4 address.

  • 1
    You can trust netstat to give you correct information. If the endpoints of a TCP connect disagree on which IP addresses are in use, it means there is a NAT somewhere between the two. You need to include the output of iptables-save in the question. – kasperd Dec 29 '15 at 9:59
  • I don't think this question has anything to do with IPv6. If you perform the exact same operation with an IPv4-only hostname, you should see the exact same behavior. – kasperd Dec 29 '15 at 10:01
  • Might be kernel-specific. After five years Red Hat kernels tend to be real mongrels, with code from many different kernel versions backported from many different subsystems. Try this again on CentOS 7, which is reasonably current. – Michael Hampton Dec 29 '15 at 14:55
  • 1
    @kasperd you were right! It was NAT (masquerade) translating the outbound IP! I disabled NAT with iptables -t nat -F and the problem went away. I had tested this with ipv4/ipv6 combo hostnames and ipv4-only hostnames that I have, and it did always ignore the outbound IP specifier on the combo hostname connections. So weird! That made no logical sense, but that was the case. I also just removed this line from my iptables -A POSTROUTING -j MASQUERADE (not sure why it was there) and that seemed to fix the problem permanently. So it was NAT causing the problem! Thank you, @kasperd !!! – Crash Override Dec 29 '15 at 15:32
  • @CrashOverride I have combined parts of my first comment and your update into an answer. – kasperd Dec 29 '15 at 17:41
3

You can trust netstat to give you correct information about the IP addresses (at least as long as -n is used).

If the endpoints of a TCP connection disagree on which IP addresses are in use, it means there is a NAT somewhere between the two.

From the additional information provided in the comments we learned that in this particular case a superfluous iptables rule -A POSTROUTING -j MASQUERADE was the cause of the problem.

  • Thanks @kasperd . Confirmed the outbound IP is being set correctly since I removed the iptables masquerade setting. You were right, netstat can be trusted. If the IP the receiving-end sees me connecting from differs from what netstat says I am connecting from, it likely would be a NAT or iptables masquerade causing the IP translation. – Crash Override Jan 2 '16 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.