Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We recently set up a new Ubuntu 12.04LTS server on our network. It's not fully configured so it's not doing much beyond sshd and a default apache2 install. But this evening it appears to have crashed. It wasn't responding to the network or the keyboard. But the worst part is, it took down the entire network.

My knowledge of the network stack below OSI layer 3 is very limited, so the rest confuses me. When this machine was physically connected to the network, no other machine could connect to the outside internet. When things were broken, running arp showed that our gateway's IP address (10.0.1.1) was listed as "invalid." Unplugging the server from the network fixed the problem, and plugging it back in broke it again. So the crashed server was advertising itself as owning the gateway's IP address?

There's nothing at all in syslog during the time when it was causing problems. Any ideas about how to figure out what went wrong or what we can do to prevent it from happening again? I'm hesitant to even put the machine back on the network right now.

Update **

It crashed again, and I ran tcpdump -penn arp (thanks bahamat!) for several minutes and got this... (timestamps and duplicate lines removed)

00:1e:65:f8:dc:24 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.0.1.1 tell 10.0.2.191, length 46
00:1e:65:f8:dc:24 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.0.1.44 tell 10.0.2.191, length 46
60:d8:19:d4:71:d6 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.0.1.1 tell 10.0.2.125, length 46
d4:9a:20:04:e9:78 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.1 tell 192.168.1.100, length 28

Update 2 **

When the network is functioning properly, arping -c4 10.0.1.1 returns this:

ARPING 10.0.1.1
60 bytes from c0:c1:c0:77:25:8e (10.0.1.1): index=0 time=267.982 usec
60 bytes from c0:c1:c0:77:25:8e (10.0.1.1): index=1 time=422.955 usec
60 bytes from c0:c1:c0:77:25:8e (10.0.1.1): index=2 time=299.215 usec
60 bytes from c0:c1:c0:77:25:8e (10.0.1.1): index=3 time=366.926 usec

--- 10.0.1.1 statistics ---
4 packets transmitted, 4 packets received,   0% unanswered (0 extra)

When the bad server is plugged in, arping -c4 10.0.1.1 returns:

ARPING 10.0.1.1

--- 10.0.1.1 statistics ---
4 packets transmitted, 0 packets received, 100% unanswered (0 extra)

Context **

  • 10.0.x.x is the main subnet.
  • 10.0.1.1 is the main internet gateway
  • 10.0.1.44 is a printer
  • 10.0.2.* devices are all laptops / workstations
  • I have no idea what's using the 192.168.x.x subnet -- your guesses are at least as good as mine. A VM on a workstation? A misconfigured WAP? Somebody re-sharing wifi? A machine that failed to DHCP?
  • The offending ubuntu server's MAC address ends in cd:80 so isn't listed in the dump. It should DHCP to 10.0.3.3

Thanks for any help. This ARP stuff is all voodoo to me. Packets just go to IP addresses, right? ;)

share|improve this question
2  
On another host run tcpdump -penn arp then plug the machine in and see what's going on. If you describe what's happening someone might be able to provide more help. –  bahamat Jul 9 '12 at 4:50
    
tcpdump output added. Thanks bahamat! –  Leopd Jul 9 '12 at 15:44
    
Could you add the output of arping -c 4 to the question? The tcpdump log shows that there are requests but not who is answering. And maybe the output of arp -amight be helpfull. –  Christopher Perrin Jul 9 '12 at 17:46
1  
I'm sorry I meant arping -c4 <gateway>. –  Christopher Perrin Jul 10 '12 at 17:55
    
Okay, I tried that... –  Leopd Jul 11 '12 at 21:37
show 1 more comment

2 Answers

Just had the exact same issue. All of a sudden most of my network went down. The only part still working was WiFi and I could only connect to the router, could not reach WAN and none of the wired LAN computers answered my pings. After rebooting the router several times to no avail I resorted to unplugging all ethernet cables. All of sudden it worked again, reconnected the cables and everything went down. After a bit of trial and error I found the culprit; my headless Ubuntu 12.04 server. I could kill the network by plugging it in and revive it by unhooking it. Eventually I resorted to pull the power. When it came back up it played nicely, I checked syslog and to my great surprise there was absolutely nothing there;

Sep 17 21:21:44 *** Normal event occuring
Sep 17 21:22:16 *** Normal event occuring
Sep 17 21:22:48 *** Normal event occuring
Sep 17 21:23:20 *** Normal event occuring
Sep 17 22:45:36 Atlas kernel: imklog 5.8.6, log source = /proc/kmsg started.
Sep 17 22:45:36 Atlas rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="1048" x-info="http://www.rsyslog.com"] start
Sep 17 22:45:36 Atlas rsyslogd: rsyslogd's groupid changed to 103
Sep 17 22:45:36 Atlas rsyslogd: rsyslogd's userid changed to 101

Strange indeed and kind of worrying. Not only did my server that has been stable since I first fired it up go down but it managed to bring the rest of the network with it.

share|improve this answer
add comment

Well, I can tell you that the machine on 192.168.1.x is a MAC address issued to Apple.

Are you receiving the ARP requests on the gateway itself? What about dumping traffic from the switch? It's sounding like the Ubuntu machine might be getting sent ARPs that it shouldn't and it could be confusing the switch.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.