3

Hi I'm not a sysadmin but I have two FC11 boxes that are behaving strangely after the circuit breaker tripped (overload elsewhere - fixed). A can ping B but when B (or any other node on network) pings A I get 'destination host unreachable.

all nodes are connected via a managed L2 switch so its just a simple A---switch---B to get from A to B.

The firewall on B has been turned off for this test (prior to that it did allow ICMP inbound anyway, and it worked fine before the circuit tripped).

Its not just ping that doesnt work on B, I cant SSH to B either.

B is configured as software RAID 1 and I did check the array and it appears to be OK after the power came back

Notable history: When I restarted after resetting the CB, it took a LONG time for the logon screen to appear, there was lots of HD activity, and after it started up it only showed 'Other' as a valid user, my normal account was not listed. After rebooting it reappeared and the boot speed as back to normal but I cant access any services or ping B.

I'm not a sysadmin so I dont know where to really start looking besides the obvious (firewall, network connection, test other clients, restart comp, SMART attributes OK)

Appreciate if anyone can suggest where I should start looking.

EDIT:

Solved! Thanks to Wayne's list I've found the culprit, the L2 managed switch configuration got corrupted (I have no idea how?!) I switched ports around between the different clients and confirmed it.

It was so badly corruped I had to use the serial cable + minicom to reset it. It deleted all the trusted hosts aside from one ip that I wasnt connecting from (hence I was unable to telnet/web into it) and decided to block traffic in one way between A/B ports. Finding and fixing it was quite a frustrating (but educational) experience! I would never have suspected the managed switch would be capable of doing this.

Its a D-link DES-3000 series switch. I'm now wondering if it was a freak incident due to a surge perhaps, or if the switch is broken in some other bad way.... Thoughts? Should I replace it?

Improve this question
1
  • Maybe you could append the active firewall rules & policies to your question? That's the output of "iptables -v -L" and "iptables -v -t nat -L".
    – conny
    Nov 17, 2009 at 10:19

4 Answers 4

Reset to default
5

Lets first work out where the problem resides.

  1. Can you access the required services on B when logged into B?
    1. You will need to restore from backup as it looks like OS has become corrupt
  2. Can you acccess the required services on A when logged into A?
    1. You will need to restore from backup as it looks like OS has become corrupt
  3. Can you flush and reset iptables rules on A to ensure not a corrupt firewall: iptables -X && iptables -F
  4. Can you try and access services on A from B?
  5. Can you use telnet on B to telnet to service port on A? eg: telnet A 80 or telnet A 25
  6. Can you flush and reset iptables rules on B to ensure not a corrupt firewall: iptables -X && iptables -F
  7. Can you try and access services on B from A?
  8. Can you use telnet on A to telnet to service port on B? eg: telnet B 80 or telnet B 25
  9. Can you restart the firewall on A and re-test from B?
  10. Can you restart the firewall on B and re-test from B?

With the answers to these questions we can start deducing where the problem lies at the moment it is pure speculation where the problem lies without some real analysis and data to collaborate what the cause of the issue is.

Also worth noting:

  1. Have you verified your switch is ok?
  2. Have you tried other machines on this switch?
  3. Have you tried other ports?
  4. Have you verified the switch's configuration?
Improve this answer
2
  • Your checklist saved the day for me, thanks. Comprehensive and helped me to narrow down the cause. In future I wont be so lost! Now I need to figure out what to do about the switch.
    – Ron
    Nov 17, 2009 at 11:48
  • We have all been in the assumption part of things before. I have learnt from those mistakes as the assumption is usually guaranteed to be wrong! I thought it was the switch, but needed you to test things before pointing a finger at the culprit.
    – Wayne
    Nov 17, 2009 at 12:05
0

Wayne's list is pretty comprehensive, but if you can stand the downtime/interruption, you may also want to change the cables connecting A and B to the switch; I've personally seen cabling fail after a power cut, despite the devices at both ends of the cabling still being OK. And if you have a spare switch (or even hub) lying around, you could temporarily use it to replace the switch in your equation; if everything starts working after that, you know where your problem is.

Improve this answer
1
  • Never knew that was possible, will remember that. I did end up switching the cables into different ports to confirm that the switch was the fault as per edit.
    – Ron
    Nov 17, 2009 at 11:47
0

The fact that A can ping B, but B can't ping A is very odd. Some questions:

  1. Do all the machines in question have a single network card each connected to the L2 switch?
  2. Are you pinging by name or by IP?
  3. When you ping (in both directions), do valid ARP entries for the destination host appear in the source host's routing table?
  4. Do you have a log in to the switch? If so, can you confirm that all the ports in question are all in the correct VLAN?

If I had to guess, I suspect that host A has two NICs with two different IPs, and that its default interface is different to the one that the other hosts are using to ping it. This is pure speculation, however; are you able to provide the output of 'ifconfig -a' and 'netstat -rnv' from host A?

Improve this answer
1
  • only 1 NIC each, see my edit. managed switch was the fault.
    – Ron
    Nov 17, 2009 at 11:45
0

This also happens when you are connecting with router and using virtual machine inside your existing OS which uses NAT, because of NAT create further private ip address from already created ip address.

So when you try to ping a machine connected with same router will be pingable and when you try to ping this virtual machine from any machine connected to this router will not be able to ping

Because

In this scenario your machine which has virtual machine is working like a router every ping request from inside made by its childs(Virtual machine/s) can go out and every ping request made by other clients connect with that router to ping you virtual machine will fail because this require a Bridge connection. so you need to make a Bridge connection in your virtual machine if you are facing same problem because the Bridge does not further create private ip of existent private ip ,it creates a new private ip

Example

Suppose there are two machines connect with the same router has an ip

A= 192.168.2.1 (virtual machine installed)

B=192.168.2.2

then after using Bridge in machine A the virtual machine will get an ip address

C=192.168.2.3

So this will be pingable by all machines connected with the same router and this machine too.

Improve this answer

You must log in to answer this question.