Recently my gentoo server began dropping all network connections. Since it's a headless server hidden in a closet I don't have any other means of logging on, so I'm unable to get in when this happens. The only way to restore connection is to use the reset-button and reboot the machine.
There are no messages logged from the kernel or anywhere else when it happens, atleast not in a way that gets saved to disk for inspection after the reboot.
Some short bits of information about the system:
- Regular linux kernel 2.6.35
- Running x86, on a Pentium D with 1GB RAM
- On-board networking and an external card, of which only the on-board has a cable plugged in
- On-board is VIA Technologies, Inc. VT6102 [Rhine-II] (rev 7c)
- External is Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)
- Loaded modules via-rhine, 8139cp, 8139too (all of which has been on the system for some time)
Recent changes:
- I recently migrated to an OpenRC based layout, following the gentoo migration guide.
- I installed brscan4 and brscan-skey (Brother scanning software/drivers)
- Upgraded libieee1284 to 0.2.11-r1
I have not been able to find any pattern to when it happens, and I have not been able to determine if it's only the network that dies, or if everything else stops aswell.
I need help figuring out what is going on, so I need tips to solve these problems:
- How can I determine if it is just the networking, or if everything dies?
- Assuming it is the network, how can I get some logging enabled to tell me what is happening?
I'll happily accept both answers to these questions, suggestions as to what might be wrong, or other questions that might get me closer to figuring out what is happening.
Update:
I uninstalled brscan4 and brscan-skey as those were the latest changes done, and I still had a hang happen, so they are eliminated from the possible causes. I also made syslog mark the log every minute, to have an easy way to track when things stop on the server. Turns out when the net dropped, syslog stopped logging immediately, so it seems it's not just the network, it's a full system hang.
I also built a fresh kernel, using the genkernel tool, so after the last hang I am now running on 2.6.38-gentoo-r6, hoping this will somehow solve the problem.
Update 2:
Upgraded kernel did not help.
I have now been watching lm_sensor output for any changes in temperature (or anything else) prior to a hang, but with the last three hangs there has been no noticeable changes in temperature before or after a hang, so that theory seems to be a bust.
A new trend is that the hang occurs while watching movies using MediaTomb as a DLNA-server. However, there is no direct connection here, as we can watch for several hours without a problem, turn off the TV, and then a couple hours later when we come back and want to watch something it hangs after a couple minutes, three times in a row. But the last ten hangs have all been while playing a movie, with numerous hours of other use not causing any problems.
Update 3:
Tried switching to use the other network-card, without much success.
Update 4:
Finally some progress? After sufficient amount of hangs, it was now time for a forced disk check, which proceeded fine for a while, but then it hung again. I'm thinking this might be indication that the problem lies with the disk, or the disk-controller, as there's not much else going on at this point in the boot-process?
Update 5:
As suggested, I ran memtest86+ overnight, unfortunately without finding any errors.
Update 6:
Some more testing today reveals the following:
- Create a 250MB file from
/dev/urandom
(usingdd
) on/dev/sda1
: No problem - Create a 250MB file from
/dev/urandom
on/dev/sdb3
: No problem - Copy a 250MB file from
/dev/sdb3
to/dev/sda1
: No problem - Copy a 250MB file from
/dev/sda1
to/dev/sdb3
: HANG (almost) instantly!
I will be testing this further, especially since many hangs seems to have happened while only reading from /dev/sdb3, while that doesn't seem to cause a problem in this test.
Any suggestions as to what could be causing this, or troubleshooting tips to try and find out?
smartctl -A
to get an idea of the disk's health. It might be damaged and need to be replaced. I'm also suspicious that you might be seeing a filesystem panic; you can install LKCD or crashdump to get the kernel panic output saved to a file for analysis.