Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Hi I'm not a sysadmin but I have two FC11 boxes that are behaving strangely after the circuit breaker tripped (overload elsewhere - fixed). A can ping B but when B (or any other node on network) pings A I get 'destination host unreachable.

all nodes are connected via a managed L2 switch so its just a simple A---switch---B to get from A to B.

The firewall on B has been turned off for this test (prior to that it did allow ICMP inbound anyway, and it worked fine before the circuit tripped).

Its not just ping that doesnt work on B, I cant SSH to B either.

B is configured as software RAID 1 and I did check the array and it appears to be OK after the power came back

Notable history: When I restarted after resetting the CB, it took a LONG time for the logon screen to appear, there was lots of HD activity, and after it started up it only showed 'Other' as a valid user, my normal account was not listed. After rebooting it reappeared and the boot speed as back to normal but I cant access any services or ping B.

I'm not a sysadmin so I dont know where to really start looking besides the obvious (firewall, network connection, test other clients, restart comp, SMART attributes OK)

Appreciate if anyone can suggest where I should start looking.

EDIT:

Solved! Thanks to Wayne's list I've found the culprit, the L2 managed switch configuration got corrupted (I have no idea how?!) I switched ports around between the different clients and confirmed it.

It was so badly corruped I had to use the serial cable + minicom to reset it. It deleted all the trusted hosts aside from one ip that I wasnt connecting from (hence I was unable to telnet/web into it) and decided to block traffic in one way between A/B ports. Finding and fixing it was quite a frustrating (but educational) experience! I would never have suspected the managed switch would be capable of doing this.

Its a D-link DES-3000 series switch. I'm now wondering if it was a freak incident due to a surge perhaps, or if the switch is broken in some other bad way.... Thoughts? Should I replace it?

share|improve this question
    
Maybe you could append the active firewall rules & policies to your question? That's the output of "iptables -v -L" and "iptables -v -t nat -L". –  conny Nov 17 '09 at 10:19
add comment

3 Answers

up vote 5 down vote accepted

Lets first work out where the problem resides.

  1. Can you access the required services on B when logged into B?
    1. You will need to restore from backup as it looks like OS has become corrupt
  2. Can you acccess the required services on A when logged into A?
    1. You will need to restore from backup as it looks like OS has become corrupt
  3. Can you flush and reset iptables rules on A to ensure not a corrupt firewall: iptables -X && iptables -F
  4. Can you try and access services on A from B?
  5. Can you use telnet on B to telnet to service port on A? eg: telnet A 80 or telnet A 25
  6. Can you flush and reset iptables rules on B to ensure not a corrupt firewall: iptables -X && iptables -F
  7. Can you try and access services on B from A?
  8. Can you use telnet on A to telnet to service port on B? eg: telnet B 80 or telnet B 25
  9. Can you restart the firewall on A and re-test from B?
  10. Can you restart the firewall on B and re-test from B?

With the answers to these questions we can start deducing where the problem lies at the moment it is pure speculation where the problem lies without some real analysis and data to collaborate what the cause of the issue is.

Also worth noting:

  1. Have you verified your switch is ok?
  2. Have you tried other machines on this switch?
  3. Have you tried other ports?
  4. Have you verified the switch's configuration?
share|improve this answer
    
Your checklist saved the day for me, thanks. Comprehensive and helped me to narrow down the cause. In future I wont be so lost! Now I need to figure out what to do about the switch. –  Ron Nov 17 '09 at 11:48
    
We have all been in the assumption part of things before. I have learnt from those mistakes as the assumption is usually guaranteed to be wrong! I thought it was the switch, but needed you to test things before pointing a finger at the culprit. –  Wayne Nov 17 '09 at 12:05
add comment

Wayne's list is pretty comprehensive, but if you can stand the downtime/interruption, you may also want to change the cables connecting A and B to the switch; I've personally seen cabling fail after a power cut, despite the devices at both ends of the cabling still being OK. And if you have a spare switch (or even hub) lying around, you could temporarily use it to replace the switch in your equation; if everything starts working after that, you know where your problem is.

share|improve this answer
    
Never knew that was possible, will remember that. I did end up switching the cables into different ports to confirm that the switch was the fault as per edit. –  Ron Nov 17 '09 at 11:47
add comment

The fact that A can ping B, but B can't ping A is very odd. Some questions:

  1. Do all the machines in question have a single network card each connected to the L2 switch?
  2. Are you pinging by name or by IP?
  3. When you ping (in both directions), do valid ARP entries for the destination host appear in the source host's routing table?
  4. Do you have a log in to the switch? If so, can you confirm that all the ports in question are all in the correct VLAN?

If I had to guess, I suspect that host A has two NICs with two different IPs, and that its default interface is different to the one that the other hosts are using to ping it. This is pure speculation, however; are you able to provide the output of 'ifconfig -a' and 'netstat -rnv' from host A?

share|improve this answer
    
only 1 NIC each, see my edit. managed switch was the fault. –  Ron Nov 17 '09 at 11:45
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.