0

No idea how/why, but here's what I'm seeing. Consider 3 CentOS 5.10 boxes:

Host A  [10.1.2.7]
Host B  [10.1.2.8]
Host C  [10.1.2.9]

All of a sudden we're getting strange behavior from Host X, 10.1.2.3. Upon investigation and a bunch of telnetting to various ports, correlating with host logs, etc., we finally surmise that every host (A, B, C) "thinks" it is 10.1.2.3 as well, such that:

hostA# telnet 10.1.2.3 80

results in Host A itself receiving that TCP/IP connection (port 80 in this example) instead of connecting to the true 10.1.2.3 (Host X). Host A is not configured as such:

hostA# ifconfig -a

eth0      Link encap:Ethernet  HWaddr 00:CC:CC:ED:F3:B0
      inet addr:10.1.2.7  Bcast:10.1.2.255  Mask:255.255.255.0
      inet6 addr: fe80::20c:29ff:feed:f3b0/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:63036 errors:0 dropped:0 overruns:0 frame:0
      TX packets:12508 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:6047120 (5.7 MiB)  TX bytes:1439770 (1.3 MiB)
      Interrupt:59 Base address:0x2000

lo        Link encap:Local Loopback
      inet addr:127.0.0.1  Mask:255.0.0.0
      inet6 addr: ::1/128 Scope:Host
      UP LOOPBACK RUNNING  MTU:16436  Metric:1
      RX packets:28378 errors:0 dropped:0 overruns:0 frame:0
      TX packets:28378 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:2410342 (2.2 MiB)  TX bytes:2410342 (2.2 MiB)

sit0      Link encap:IPv6-in-IPv4
      NOARP  MTU:1480  Metric:1
      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

I'm still suspicious of something, somehow, ARP-related, but note that an arp -n does not even list 10.1.2.3 at all--no ARP cache entry, even after the telnet, ping, etc., of 10.1.2.3. It's just as if Host A suddenly "thinks" it is 10.1.2.3 when nothing in its configuration says so.

And, what's more...so does Host B (same exact behavior), Host C, Host D, and so on. Each one of those hosts, when testing, turns out to be connecting back to itself when directed to connect to 10.1.2.3.

Finally decided to reboot one of them just to see if there was a running kernel/ARP issue (read: pretty much desperation). Not solved, though for a few seconds or maybe a minute after reboot the host doesn't "think" it's this other IP, but then it resumes said issue.

So I have a plethora of hosts, each suddenly "thinking" it is 10.1.2.3 but can find no configuration, no ARP table, nor anything else indicating they've assumed a binding to that IP. Meanwhile, it appears that a bunch of Windows hosts on the LAN are not having this problem--they're not exhibiting this behavior (may or may not be relevant).

Even tried moving one host to a different switch in case there was a switch issue. Not solved.

In case it's relevant, another common attribute is that all these CentOS hosts are actually VMs running under VMWare Server--but not all on the same host hardware. Multiple hypervisor hosts in play.

9
  • Have you tried tcpdump or similar to capture a packet trace - especially the resolution/connecting phase? Given what you've got listed, about the only thing I can think of off the top of my head is some sort of fruity NAT going on.
    – Sobrique
    Apr 28, 2014 at 19:57
  • ifconfig is deprecated. Please give us the output of ip addr instead. And in your case, the output of ip route get 10.1.2.3 might be important. Consider also giving the output of ip route list table all.
    – BatchyX
    Apr 28, 2014 at 20:18
  • What's the output of ip neigh?
    – sciurus
    Apr 28, 2014 at 20:24
  • Sure enough tcpdump yields what ip addr notes: [root@hostA tmp]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet 10.1.2.2/32 scope global lo inet 10.1.2.3/32 scope global lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever Didn't know about ip addr--am too rusty. Sure enough, there it is--it thinks it's 10.1.2.3. No idea how.
    – Dan
    Apr 28, 2014 at 20:36
  • And [root@vmdev01 tmp]# ip route get 192.168.10.53 local 192.168.10.53 dev lo src 192.168.10.53 cache <local> mtu 16436 advmss 16396 hoplimit 64
    – Dan
    Apr 28, 2014 at 20:39

0

You must log in to answer this question.

Browse other questions tagged .