Ubuntu Server 10.04 Heavy Network Traffic causes disconnect

Question

I'm currently running a headless Ubuntu 10.04 server. Installed is the LAMP stack, Joomla, Virtualbox, phpvirtualbox, webmin and proFTP.. It resolves the IP address so I can access it remotely (either the apache2 webserver or the FTP) using DDClient. Any packages installed have been installed using apt-get. Webmin, although discouraged in Ubuntu Server, is used mostly to administer the webserver aspect. This issue also appeared when I was using Ubuntu Server 10.10.

After periods of heavy network traffic, whether local or remote, the connect drops. I'm talking specifically about the transfer of files via FTP, SCP or Samba (the latter of which I seldom use). There is no response to ping or ssh. I can't FTP to the server nor can I load the website. There are times when the server has been on for a few days and everything runs fine because I haven't accessed it much, if at all (thus not much network traffic).

I've gone through a few hardware changes although I don't believe this has cause the issue: this has been happening long before I made any changes. At first I thought it was my ISP-provided router blocking traffic because of some kind of misconfiguration (perhaps assuming it was some kind of DoS attack). I've changed routers and still found no success. I've checked syslog, dmesg and kern.log for warnings but have uncovered none. I've ran memtest via the GRUB2 menu at boot and once it turned up 4 errors. I ran again with individual sticks of RAM in various slots and everything turned up fine. I've looked through the BIOS settings and everything looks fine. I've tried unplugging unnecessary pieces of hardware (other internal hard drives, CD drives, floppy, PCI cards, etc).

Any help or tips on how I can even begin to troubleshoot this would be very much appreciated. Please note that i've only started playing with servers as a hobby so my knowledge wouldn't be the most refined. I'm comfortable with command line and have the initiative to know how to look up something I can't do. Unfortunately I can't seem to find any issues like this.

Additionally: If a solution can't be found some assistance to write a script that will cause the server to reboot automatically if, after x minutes, it gets no response to pinging somewhere like google. Admittedly that's not the cleanest solution should my internet end up going down but I can't think of what else to do.

Answer 1

If the server is completely 100% hanging, then an automated reboot script may not help: unless the reboot happens before the hang you are stuck as the hang will likely affect what-ever process is intended to call the reboot script.

A regular reboot via cron may help, if it is set regular enough to trigger before any hang, but this would be treating the symptoms and not the cause. You could trigger the reboot from another machine (if it detects the server stopping responding) but that will likely require a hardware purchase in the form of a power supply/controller that can be toggled from one machine to force another to power-cycle.

I would recommend installing some sort of monitoring tool and checking to see what happens immediately before the hangs occur (i.e. just before new connections stop being accepted). I use collectd (with a custom CGI script to graph the recorded results) for general monitoring though there are several other popular options. Running such a monitoring tool with default settings (monitoring CPU use, memory use, disk I/O, temperature readings, and so forth) will help you spot obvious things like a sudden burst of CPU activity (which may imply a bad script or a DoS situation) or creeping memory/swap use (which may imply a memory leak somewhere, or in the case of Apache and similar services a worker allocation configuration that is inappropriate for the size of machine), a sudden rise in temperature (which may imply a circulation issue, poor ventilation, or other external environmental conditions being part of the problem), and so on. If a generic issue like this is identified you can then add more detailed monitoring to zero in on a more specific cause.

Also, install and configure smartd if you have not already done so. This may help tracking the problem if it is down to a drive that is developing (or already has) a serious problem.

In any case, check the usual suspects in /var/log after a hang - you may find some clues get recorded in places like /var/log/messages and /var/log/syslog (or similar) just before the machine stops responding. If nothing on the machine itself stops when the remote connections start failing, you might have a bad network card that is hanging (but leaving the rest of the machine OK) and staying in that hung state until the machine is rebooted or power-cycled.

More specifically: your RAM tests showing some errors on one or two occasions makes either RAM or cooling quite likely culprits. You could have some "slightly" dodgy RAM that usually works and passes tests but very occasionally flips bits and causes problems, or you could have a RAM issue that is temperature sensitive (all is fine until the heat hits a certain point), or it could be a more general heat/cooling issue. Your CPU or other core chips could also be experiencing heat problems that would result in similar intermittent effects.

Answer 2

This may not be the reason for you but I've been seeing this in 10.04 LTS for some time when using dhcp. However, when setting the interface address as a static one, the problem goes away.

I know it has been an issue with 10.04 LTS because I've seen it occur on at least 1 PC and 2 servers with Intel NIC's. I should also point out that this issue seems to be resolved for me with the latest build of Ubuntu 10.04 LTS server. I believe it is build 3 or 4 of that.

https://askubuntu.com/questions/102910/ubuntu-server-10-04-lts-xen-intermittent-networking

For now make sure you are not using DHCP to assign the address, even if the DHCP server is set to always give out the same address. Rather, set it statically in /etc/network/interfaces

Answer 3

Similar issue w/ Apache/PHP on RHEL5.x: hanging and console/ssh access impossible. Log /var/log/messages reported "[...] httpd invoked oom-killer [...]"

The solution was adding more memory, enabling KDump w/ panic_on_oom and creating more appropriate Apache/PHP process limits. No issues since. The KDump w/ panic_on_oom automatically restarted the system preventing a hang. However the real fix was limiting Apache/PHP. The defaults, esp. for PHP, were too wide open and not secure. There are numerous online resources for securing PHP, so I will not try and recreate one here.

Answer 4

I believe it may be due to the services heavily taxing your system resources. Another possible solution may be to look at how many possible connections you can have to your web server and/or how many hosts can use the router.