I'm not sure "honor" is the right word for this, but it's the best I could come up with. I have a scenario where I have two servers on the same network. They have primary and secondary IPs, all on the same subnet. For the sake of discussion, they look like this:
server1 eth0 172.16.45.3/24 server1-A eth0:11 172.16.45.21/27 server1-B eth0:12 172.16.45.22/27 server2 eth0 172.16.45.4/27
Yes, server1 is set to /24, and yes it's a mistake.
I noticed this problem because connections from server1->server2 had a source IP of 172.16.45.21 instead of 172.16.45.3. Since the app originating the connection does not specify a source IP, I was shocked that it wasn't using 172.16.45.3.
That's when I noticed the incorrect netmask. Since the target IP is in a known smaller network, it uses an IP from the same /27 instead of the IP it thinks is from a /24. Oops.
So, I fixed the netmask on server1:eth0 by running the following command:
ifconfig eth0 netmask 255.255.255.224
ifconfig seemed happy now too:
eth0 Link encap:Ethernet HWaddr 00:22:19:54:EF:11 inet addr:172.16.45.3 Bcast:172.16.45.31 Mask:255.255.255.224 inet6 addr: fe80::222:19ff:fe54:ef11/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1085587580 errors:0 dropped:1355 overruns:0 frame:0 TX packets:1208356392 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:365708046601 (340.5 GiB) TX bytes:667099868812 (621.2 GiB) Interrupt:169 Memory:f8000000-f8012100
Also, the routing table cleaned itself up.
server1 0 /home/jj33 ># route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 172.16.45.0 0.0.0.0 255.255.255.224 U 0 0 0 eth0 172.16.45.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 0.0.0.0 172.16.45.1 0.0.0.0 UG 0 0 0 eth0
server1 0 /home/jj33 ># route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 172.16.45.0 0.0.0.0 255.255.255.224 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0 0.0.0.0 172.16.45.1 0.0.0.0 UG 0 0 0 eth0
The only problem is, after all of that, the OS still seems to choose 172.16.45.21 as the source address for outbound connections to the same network (SMTP doesn't directly figure into this problem, just a convenient way to show the source IP of a connection):
server1 0 /home/jj33 ># telnet server2 25 Trying 172.16.45.4... Connected to 172.16.45.4. Escape character is '^]'. 220 server2.example.com ESMTP mailer ready at Wed, 23 Dec 2009 12:18:28 -0600 ehlo foo 250-server2.qcommcorp.com Hello server1-A.example.com [172.16.45.21] 250 HELP
(in case it's not obvious, I would expect the mailer to say "Hello server1.example.com [172.16.45.3]" in response to my ehlo if everything were working properly).
So, now my question. How can I get my OS to take notice of the fact that the netmask on eth0 has changed such that it's a better choice for outbound connections to my local /27? I assume restarting the server or restarting networking services would do it, but I would have to wait a week until my next maintenance window and it seems like something I could do without interrupting service (this is a production system and this improper source IP is a small, tangential problem - the core app is working well).
Any help greatly appreciated. Thanks!
So, this problem drew more attention than I was expecting and I ended up getting permission to fail the application over to the standby silo and restart networking services on the affected server outside of our standard window, meaning I can't test any of the theories below.
In general though I believe Juliano's response covered the most detail. I didn't copy and paste it but in playing with ip it generally seemed to corroborate what he posited.
Also, enough people piled on about using ip in preference to ifconfig that I spent some time playing with it and tip my hat to you all, I certainly should be using ip. Thanks for the pointers.