Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have several servers that are randomly loses the ability to communicate out to other nodes on the local network and internet. However websites are being served fine and we still have ssh access.

A reboot seems to fix the problem for a few days. The servers are running 10.5.X and are fully up to date with software updates.

We currently have the problem as described on one box. E.g., trying to do a curl google.com times-out, SMTP out is not happening, but I can SSH in, and Apache is serving web pages with no problem.

If I disable the firewall with sudo ipfw disable firewall, outgoing traffic resumes. If I then re-enable the firewall with sudo ipfw enable firewall, the problem reappears.

Firewall rules:

$ sudo ipfw show            
00001  1123672   159015812 allow udp from any 626 to any dst-port 626
01000 70125526 26073450472 allow ip from any to any via lo0
01010        0           0 deny ip from any to 127.0.0.0/8
01020        0           0 deny ip from 224.0.0.0/4 to any in
01030        0           0 deny tcp from any to 224.0.0.0/4 in
12301   904047   139023698 allow tcp from any to any out
12302    63538    11694670 allow tcp from any to any dst-port 22
12302        0           0 allow udp from any to any dst-port 22
12303   304700    44969494 allow udp from any to any out keep-state
12304        0           0 allow tcp from any to any dst-port 53 out keep-state
12304        0           0 allow udp from any to any dst-port 53 out keep-state
12304        0           0 allow tcp from any to any dst-port 53 out keep-state
12306      171       10944 allow tcp from any to any dst-port 311
12307     2167      143175 allow udp from any to any dst-port 626
12308     1604       98058 allow icmp from any to any icmptypes 8
12309     1604       98058 allow icmp from any to any icmptypes 0
12311   228466    17139842 allow tcp from any to any dst-port 80
12312     3349      616527 allow tcp from any to any dst-port 443
12313      386       18279 allow tcp from any to any dst-port 3283,5900
12313        4         130 allow udp from any to any dst-port 3283,5900
65534   885514    49998498 deny ip from any to any
65535        0           0 allow ip from any to any

Does anyone have any opinions? We think it is probably an ISP router, but we are guessing.

share|improve this question
add comment

3 Answers

It sounds like a routing issues on the internal network. Some more insight into your topology would help immensely .

I'm making a leap here but it almost sounds like the system might lose the internal connection, and the routing table gets screwy.

Do a netstat -r while you know the system is working as expected and then when it fails. We're interested in the "default" route (example below).

netstat -r
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            192.168.1.1        UGSc           53        0     en1

If the route looks the same, it could be an upstream (as in your local network, not the internet) router/switch sending packets the wrong place.

share|improve this answer
add comment

If it was an ISP router, why would that affect your ability to talk to nodes on the local network? Otherwise, I would run wireshark and check the logs.

share|improve this answer
    
Router/Switch. The assumption was we haven't changed the configuration of the box at all so it would need to be an external source. –  jonoabroad Nov 20 '09 at 3:56
    
But will try wireshark. Thank you. –  jonoabroad Nov 20 '09 at 3:57
add comment

Topology info would be helpful, but this sounds to me like a switch MAC-address or ARP table exhaustion issue. Or a NAT table exhaustion where applicable. A link reset would clear parts of it. Instead of a reboot, disconnect the ethernet cable for 2 minutes, then reconnect, and see if that restores it for a while like the reboot did.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.