Why am I only able to connect to my server when generating traffic from it?

Question

UPDATE - Because of the infrastructure changes from the acquisition, there was an IP address conflict. Once that was identified, the behavior stopped, as traffic could consistently be routed to the correct location. Thank you all for your help.

I have a Virtualized Centos 6.2 FINAL server. The intent of the machine is to run Jenkins on Tomcat for automated builds. As my company was acquired, the network infrastructure changed and so some of the configuration was changed as well. I can go into the history more if needed, but for now, I'll focus on my actual symptoms.

1. The machine, if attempted to be interacted with via network over ssh or http, does not respond typically.
2. The machine, if logged into directly, will interact with machines. However, it seems to take a bit to finally start conversing. That is, when I run a traceroute from the vm to my local machine, it takes a few iterations. Same with ping - it can take up to 6 iterations before the packets begin to make it through.
3. Once the machine is conversing with other machines, outside contact is achievable. I can ssh to it, and the Jenkins application will respond over http.
4. The communication will eventually cease however and return to the initial first state where outside contact is not possible, until I directly interact with the machine to cause it to communicate with the outside world (2).

I'm at a bit of a loss to understand what to look into from a diagnostic perspective. As I said earlier, there was some history here. The machine originally had one network adapter with an ip address assigned via DHCP. Then one additional network adapter was added, and the ip address was assigned statically. Now there is only one network adapter with the ip address assigned statically.

I've looked at other threads out on the Internet that talk about various commands that can help diagnose, but I'm not sure how to read them.

I imagine this isn't enough information to make a diagnosis (though would be grateful if it was!) so I will appreciate further reading materials, steps to take, or clarifying questions.

Thanks so much for your time.

Answer 1

A possibility beyond simple MAC table filling is a gratuitous ARP collision.

The case I'm thinking of requires two things:

1. A bad netmask, probably on the Jenkins box.
2. One of:
   1. More than one device configured for gratuitous ARP on the Jenkins box subnet, one of which is stateful (perhaps a firewall or a load-balancer).
   2. The device configured for G-ARP is not the device you expect to be the default gateway.

How Gratuitous ARP is supposed to work

In the initial state when the Jenkins box hasn't been interacted with and all the various router/switch/arp tables are empty, if your mis-configured Jenkins box attempts to access a host it considers local due to the bad netmask but is actually remote the device configured for Gratuitous ARP will lie to it and say it's the IP address in question and quietly route it where it needs to go.

Multiple G-ARP device case

In this case, there are two devices sending G-ARP packets. The failure mode here is:

1. Jenkins sends an ARP request for an off-subnet address.
2. The first G-ARP device replies that it has that address.
3. Jenkins sends the SYN packet to the remote device.
4. The remote device ACKs the connection and is received by Jenkins.
5. Before the SYN-ACK be issued, the 2nd G-ARP device also send an ARP reply that it has that address. Jenkins dutifully updates its local ARP table.
6. The SYN-ACK packet goes through the 2nd G-ARP device. Unfortunately, since it never saw the initial SYN packet it isn't aware of the connection so it drops it on the floor.
7. The connection never opens.

This is visible on both hosts through watching the socket states in netstat. And quite visible if you grab packets on the Jenkins box.

Connections TO it work because they're coming through the 2nd G-ARP device, and by so doing update the Jenkins box ARP table with the 'right' device to pass packets through. So long as that connection is up and passing traffic, outbound traffic will go the right way.

Wrong device responds to G-ARP

This is a variation on the above. Only one device is configured for G-ARP, but it isn't the right one. The G-ARP is almost definitely a bad configuration.

1. Jenkins sends an ARP request for an off-subnet address.
2. The G-ARP device replies that it has that addresss.
3. Jenkins sends the SYN packet to that device.
4. The device isn't actually a router, or isn't configured to pass packets like that, so drops it on the floor.

As with the above, when you establish a connection it sets up the ARP table to pass things the right way so long as the connection is up.

---

The easiest fix is to verify that your netmask on that box is correct for the subnet it's on as that bypasses the whole G-ARP problem.

The other possibility is that there is something hinky with the default gateway you've configured. If it's what everything else is, you should be seeing the same behavior for anything using that gateway; if that's the case, verify that the same problem doesn't happen for other VMs on the same host.

Answer 2

Review your switches' config.

It's a hunch, but it seems to me that your switches are hit with much traffic, filling up their MAC tables and making them forget your server's MAC address/port.

Combined with STP, it can take up to 30 seconds to make your traffic pass through the switch again.