Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Recently my gentoo server began dropping all network connections. Since it's a headless server hidden in a closet I don't have any other means of logging on, so I'm unable to get in when this happens. The only way to restore connection is to use the reset-button and reboot the machine.

There are no messages logged from the kernel or anywhere else when it happens, atleast not in a way that gets saved to disk for inspection after the reboot.

Some short bits of information about the system:

  • Regular linux kernel 2.6.35
  • Running x86, on a Pentium D with 1GB RAM
  • On-board networking and an external card, of which only the on-board has a cable plugged in
    • On-board is VIA Technologies, Inc. VT6102 [Rhine-II] (rev 7c)
    • External is Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)
    • Loaded modules via-rhine, 8139cp, 8139too (all of which has been on the system for some time)

Recent changes:

  • I recently migrated to an OpenRC based layout, following the gentoo migration guide.
  • I installed brscan4 and brscan-skey (Brother scanning software/drivers)
  • Upgraded libieee1284 to 0.2.11-r1

I have not been able to find any pattern to when it happens, and I have not been able to determine if it's only the network that dies, or if everything else stops aswell.

I need help figuring out what is going on, so I need tips to solve these problems:

  1. How can I determine if it is just the networking, or if everything dies?
  2. Assuming it is the network, how can I get some logging enabled to tell me what is happening?

I'll happily accept both answers to these questions, suggestions as to what might be wrong, or other questions that might get me closer to figuring out what is happening.

Update:

I uninstalled brscan4 and brscan-skey as those were the latest changes done, and I still had a hang happen, so they are eliminated from the possible causes. I also made syslog mark the log every minute, to have an easy way to track when things stop on the server. Turns out when the net dropped, syslog stopped logging immediately, so it seems it's not just the network, it's a full system hang.

I also built a fresh kernel, using the genkernel tool, so after the last hang I am now running on 2.6.38-gentoo-r6, hoping this will somehow solve the problem.

Update 2:

Upgraded kernel did not help.

I have now been watching lm_sensor output for any changes in temperature (or anything else) prior to a hang, but with the last three hangs there has been no noticeable changes in temperature before or after a hang, so that theory seems to be a bust.

A new trend is that the hang occurs while watching movies using MediaTomb as a DLNA-server. However, there is no direct connection here, as we can watch for several hours without a problem, turn off the TV, and then a couple hours later when we come back and want to watch something it hangs after a couple minutes, three times in a row. But the last ten hangs have all been while playing a movie, with numerous hours of other use not causing any problems.

Update 3:

Tried switching to use the other network-card, without much success.

Update 4:

Finally some progress? After sufficient amount of hangs, it was now time for a forced disk check, which proceeded fine for a while, but then it hung again. I'm thinking this might be indication that the problem lies with the disk, or the disk-controller, as there's not much else going on at this point in the boot-process?

Update 5:

As suggested, I ran memtest86+ overnight, unfortunately without finding any errors.

Update 6:

Some more testing today reveals the following:

  • Create a 250MB file from /dev/urandom (using dd) on /dev/sda1: No problem
  • Create a 250MB file from /dev/urandom on /dev/sdb3: No problem
  • Copy a 250MB file from /dev/sdb3 to /dev/sda1: No problem
  • Copy a 250MB file from /dev/sda1 to /dev/sdb3: HANG (almost) instantly!

I will be testing this further, especially since many hangs seems to have happened while only reading from /dev/sdb3, while that doesn't seem to cause a problem in this test.

Any suggestions as to what could be causing this, or troubleshooting tips to try and find out?

share|improve this question
    
What type of headless server is this? Does it at least have USB and Serial connectors someplace? –  Rilindo Jul 1 '11 at 21:25
    
It's a regular old PC, so technically I can attach a monitor and a keyboard, but neither are connected at the moment. As a regular old PC, it has both USB and serial connectors. –  Epcylon Jul 1 '11 at 21:53
    
An old pc running in a closet? Are you sure it's not thermal related? Try leaving the closet door open for a while –  Joris Jul 2 '11 at 11:11
    
I have considered that, it's certainly a possibility, but it has been in the closet for well over three years without any problems earlier. The hangs have also happened with the door open. I'd like to remove the possibility though, so any tips on how to look for signs of thermal problems? –  Epcylon Jul 2 '11 at 11:16
    
If the hard disk supports SMART, you could install the smart tools and run smartctl -A to get an idea of the disk's health. It might be damaged and need to be replaced. I'm also suspicious that you might be seeing a filesystem panic; you can install LKCD or crashdump to get the kernel panic output saved to a file for analysis. –  Handyman5 Jul 3 '11 at 19:15
add comment

4 Answers

You might want to try configuring some sort of logging process on the server that reports out, so that you can tell if it's still alive when the network connections drop. This could be something as simple as a cron job that wgets a remote URL (which you can then monitor the traffic log of), to setting up a remote syslog server that captures all log traffic from the box.

You could also schedule a restart of the networking system with cron (/etc/init.d/net.eth0 restart once an hour or so); that way, if you lost connection because of some problem with the network interface, it might be cleared up by a restart without bouncing the entire box.

The next thing I'd do, though, would be the following:

  1. Set up a cron job on your desktop to ping the box every minute so you know immediately when it stops responding. (Perhaps have it output to a log with a timestamp each ping so you don't have to write it down somewhere else.)
  2. Wait ten minutes.
  3. Go reboot the server.
  4. Connect to the server and look at /var/log/messages (and /var/log as a whole, really) to see if anything happened on the box between the time when it stopped responding and the time when you restarted it.

If the log messages stop when the network connection does, the box is likely kernel panicking, and you can enable extra logging for that to find out why. If the log messages continue after the network connection dies, then something else is at fault. (In that case I'd be suspicious of a flaky network card or bad driver, but that's just me.) You could write up a little suite of network diagnostics, ifconfig/traceroute/etc., and run that from cron into a log file; then when you have a timeframe for the network being inactive, you can inspect the log to see what the box is seeing at that time.

UPDATE: Since it looks like the problem is a full kernel panic, the next thing I'd try is setting up lm_sensors and writing the output of the sensors command to a log every minute. That way you'll be able to see if there are any rapid or gradual changes in temperature that tend to correlate with the panics.

share|improve this answer
add comment

From the collective updates, it looks like this situation may be heading down towards two possibilities:

  • Hard disk issues
  • RAM issues

Tests to try:

  • Hard drive testing (read-only, read/write)
  • Memory testing
share|improve this answer
    
I agree, you've just about now mentioned everything in your updates, except results of running through some memtest86+ cycles. –  lkraav Jul 3 '11 at 21:19
add comment

You might try switching to or also utilizing the second NIC, if that has not been looked into already. That will assist in identifying or eliminating the primary NIC and/or its associated cabling, hardware, etc.

share|improve this answer
    
I'll give it a shot next time it happens. –  Epcylon Jul 1 '11 at 21:58
1  
You should have the second NIC up and running, so it can be used to assist with the testing/determination should/when the next occurrence takes place. –  user48838 Jul 1 '11 at 22:27
add comment

Not sure if this should be an answer or a comment.

Your description feels a bit like https://bugs.gentoo.org/show_bug.cgi?id=359671. Although some parameters of the situation seem to be different, esp. the fact that you don't even have a cable connected to realtek, I guess it can still possibly have something to do with the realtek driver's wonkyness and it's probably worth it to throw this idea out here.

So try blacklisting the realtek modules.

share|improve this answer
    
This bug seems to be happening on a 2.6.38 kernel (the one I just upgraded to), and not as early as 2.6.35 where I had my problems. Also, it seems to be related to high network activity, while the first times this happened to me, there was virtually no activity. –  Epcylon Jul 2 '11 at 11:10
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.