Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm currently running a headless Ubuntu 10.04 server. Installed is the LAMP stack, Joomla, Virtualbox, phpvirtualbox, webmin and proFTP.. It resolves the IP address so I can access it remotely (either the apache2 webserver or the FTP) using DDClient. Any packages installed have been installed using apt-get. Webmin, although discouraged in Ubuntu Server, is used mostly to administer the webserver aspect. This issue also appeared when I was using Ubuntu Server 10.10.

After periods of heavy network traffic, whether local or remote, the connect drops. I'm talking specifically about the transfer of files via FTP, SCP or Samba (the latter of which I seldom use). There is no response to ping or ssh. I can't FTP to the server nor can I load the website. There are times when the server has been on for a few days and everything runs fine because I haven't accessed it much, if at all (thus not much network traffic).

I've gone through a few hardware changes although I don't believe this has cause the issue: this has been happening long before I made any changes. At first I thought it was my ISP-provided router blocking traffic because of some kind of misconfiguration (perhaps assuming it was some kind of DoS attack). I've changed routers and still found no success. I've checked syslog, dmesg and kern.log for warnings but have uncovered none. I've ran memtest via the GRUB2 menu at boot and once it turned up 4 errors. I ran again with individual sticks of RAM in various slots and everything turned up fine. I've looked through the BIOS settings and everything looks fine. I've tried unplugging unnecessary pieces of hardware (other internal hard drives, CD drives, floppy, PCI cards, etc).

Any help or tips on how I can even begin to troubleshoot this would be very much appreciated. Please note that i've only started playing with servers as a hobby so my knowledge wouldn't be the most refined. I'm comfortable with command line and have the initiative to know how to look up something I can't do. Unfortunately I can't seem to find any issues like this.

Additionally: If a solution can't be found some assistance to write a script that will cause the server to reboot automatically if, after x minutes, it gets no response to pinging somewhere like google. Admittedly that's not the cleanest solution should my internet end up going down but I can't think of what else to do.

share|improve this question
    
I seem to be having a very similar problem on my 10.04 server at home. I'll watch this question closely, and keep you updated if i find a solution. –  Sirex Dec 19 '10 at 4:39
    
Following the advice below I've installed Collectd. I transferred more than 50GB last night without any problems. After that I began, using FTP, the transfer of approximately 130GB of files. Somewhere at 2am this morning the server froze again. I've purged webmin completely, too just in case this was an issue. Approximately at the same time the following command ran: Dec 23 02:39:01 kserver CRON[18114]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -n 200 -r -0 rm) - maybe related? –  K Vaughan Dec 23 '10 at 11:21
    
This may be coincidence. According to the information by collectd, system load, CPU usage and physical memory were all quite high around that time but reported that there was still RAM available. –  K Vaughan Dec 23 '10 at 11:24
    
I may have a solution to this. If you run "lspci" - what network card do you have ? –  Sirex Jan 8 '11 at 11:20
    
Apologies for the very slow reply, Sirex. I've been travelling a bit with limited internet access. I'll respond with this information later this evening. –  K Vaughan Jan 19 '11 at 12:52

3 Answers 3

Similar issue w/ Apache/PHP on RHEL5.x: hanging and console/ssh access impossible. Log /var/log/messages reported "[...] httpd invoked oom-killer [...]"

The solution was adding more memory, enabling KDump w/ panic_on_oom and creating more appropriate Apache/PHP process limits. No issues since. The KDump w/ panic_on_oom automatically restarted the system preventing a hang. However the real fix was limiting Apache/PHP. The defaults, esp. for PHP, were too wide open and not secure. There are numerous online resources for securing PHP, so I will not try and recreate one here.

share|improve this answer

This may not be the reason for you but I've been seeing this in 10.04 LTS for some time when using dhcp. However, when setting the interface address as a static one, the problem goes away.

I know it has been an issue with 10.04 LTS because I've seen it occur on at least 1 PC and 2 servers with Intel NIC's. I should also point out that this issue seems to be resolved for me with the latest build of Ubuntu 10.04 LTS server. I believe it is build 3 or 4 of that.

http://askubuntu.com/questions/102910/ubuntu-server-10-04-lts-xen-intermittent-networking

For now make sure you are not using DHCP to assign the address, even if the DHCP server is set to always give out the same address. Rather, set it statically in /etc/network/interfaces

share|improve this answer

If the server is completely 100% hanging, then an automated reboot script may not help: unless the reboot happens before the hang you are stuck as the hang will likely affect what-ever process is intended to call the reboot script.

A regular reboot via cron may help, if it is set regular enough to trigger before any hang, but this would be treating the symptoms and not the cause. You could trigger the reboot from another machine (if it detects the server stopping responding) but that will likely require a hardware purchase in the form of a power supply/controller that can be toggled from one machine to force another to power-cycle.

I would recommend installing some sort of monitoring tool and checking to see what happens immediately before the hangs occur (i.e. just before new connections stop being accepted). I use collectd (with a custom CGI script to graph the recorded results) for general monitoring though there are several other popular options. Running such a monitoring tool with default settings (monitoring CPU use, memory use, disk I/O, temperature readings, and so forth) will help you spot obvious things like a sudden burst of CPU activity (which may imply a bad script or a DoS situation) or creeping memory/swap use (which may imply a memory leak somewhere, or in the case of Apache and similar services a worker allocation configuration that is inappropriate for the size of machine), a sudden rise in temperature (which may imply a circulation issue, poor ventilation, or other external environmental conditions being part of the problem), and so on. If a generic issue like this is identified you can then add more detailed monitoring to zero in on a more specific cause.

Also, install and configure smartd if you have not already done so. This may help tracking the problem if it is down to a drive that is developing (or already has) a serious problem.

In any case, check the usual suspects in /var/log after a hang - you may find some clues get recorded in places like /var/log/messages and /var/log/syslog (or similar) just before the machine stops responding. If nothing on the machine itself stops when the remote connections start failing, you might have a bad network card that is hanging (but leaving the rest of the machine OK) and staying in that hung state until the machine is rebooted or power-cycled.

More specifically: your RAM tests showing some errors on one or two occasions makes either RAM or cooling quite likely culprits. You could have some "slightly" dodgy RAM that usually works and passes tests but very occasionally flips bits and causes problems, or you could have a RAM issue that is temperature sensitive (all is fine until the heat hits a certain point), or it could be a more general heat/cooling issue. Your CPU or other core chips could also be experiencing heat problems that would result in similar intermittent effects.

share|improve this answer
    
David, Thank you for taking the time to write such an informative answer. I'm in the process of installing the monitoring tools you've prescribed in the hope that I can find the cause before considering resorting to a cron script. The final comment you made regarding temperature is very plausible - sometimes the room in which I keep the server gets quite warm and heavy CPU/RAM usage will just make the server warmer. Thank you again. –  K Vaughan Dec 19 '10 at 13:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.