No idea how/why, but here's what I'm seeing. Consider 3 CentOS 5.10 boxes:
Host A [10.1.2.7]
Host B [10.1.2.8]
Host C [10.1.2.9]
All of a sudden we're getting strange behavior from Host X, 10.1.2.3. Upon investigation and a bunch of telnetting to various ports, correlating with host logs, etc., we finally surmise that every host (A, B, C) "thinks" it is 10.1.2.3 as well, such that:
hostA# telnet 10.1.2.3 80
results in Host A itself receiving that TCP/IP connection (port 80 in this example) instead of connecting to the true 10.1.2.3 (Host X). Host A is not configured as such:
hostA# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:CC:CC:ED:F3:B0
inet addr:10.1.2.7 Bcast:10.1.2.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:feed:f3b0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:63036 errors:0 dropped:0 overruns:0 frame:0
TX packets:12508 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6047120 (5.7 MiB) TX bytes:1439770 (1.3 MiB)
Interrupt:59 Base address:0x2000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:28378 errors:0 dropped:0 overruns:0 frame:0
TX packets:28378 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2410342 (2.2 MiB) TX bytes:2410342 (2.2 MiB)
sit0 Link encap:IPv6-in-IPv4
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
I'm still suspicious of something, somehow, ARP-related, but note that an arp -n does not even list 10.1.2.3 at all--no ARP cache entry, even after the telnet, ping, etc., of 10.1.2.3. It's just as if Host A suddenly "thinks" it is 10.1.2.3 when nothing in its configuration says so.
And, what's more...so does Host B (same exact behavior), Host C, Host D, and so on. Each one of those hosts, when testing, turns out to be connecting back to itself when directed to connect to 10.1.2.3.
Finally decided to reboot one of them just to see if there was a running kernel/ARP issue (read: pretty much desperation). Not solved, though for a few seconds or maybe a minute after reboot the host doesn't "think" it's this other IP, but then it resumes said issue.
So I have a plethora of hosts, each suddenly "thinking" it is 10.1.2.3 but can find no configuration, no ARP table, nor anything else indicating they've assumed a binding to that IP. Meanwhile, it appears that a bunch of Windows hosts on the LAN are not having this problem--they're not exhibiting this behavior (may or may not be relevant).
Even tried moving one host to a different switch in case there was a switch issue. Not solved.
In case it's relevant, another common attribute is that all these CentOS hosts are actually VMs running under VMWare Server--but not all on the same host hardware. Multiple hypervisor hosts in play.
tcpdump
or similar to capture a packet trace - especially the resolution/connecting phase? Given what you've got listed, about the only thing I can think of off the top of my head is some sort of fruity NAT going on.ifconfig
is deprecated. Please give us the output ofip addr
instead. And in your case, the output ofip route get 10.1.2.3
might be important. Consider also giving the output ofip route list table all
.ip neigh
?[root@hostA tmp]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet 10.1.2.2/32 scope global lo inet 10.1.2.3/32 scope global lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever
Didn't know about ip addr--am too rusty. Sure enough, there it is--it thinks it's 10.1.2.3. No idea how.[root@vmdev01 tmp]# ip route get 192.168.10.53 local 192.168.10.53 dev lo src 192.168.10.53 cache <local> mtu 16436 advmss 16396 hoplimit 64