Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Symptoms:

  • Server becomes unresponsive - Increase in load, all services stop
  • Loss of connectivity - Ping/SSH
  • Flush MySQL hosts after reboot - As MySQL refuses new connections
  • Intermittent Apache crashes
  • Generally happens early morning hours - 2 days of the week are however excluded

Changes made:

  • Updated the OS - to Ubuntu 10.04.4 LTS
  • Not sure if the MySQL server was also updated in the process
  • Current MySQL version - mysql Ver 14.14 Distrib 5.1.63, for debian-linux-gnu (x86_64) using readline 6.1
  • Updated Plesk from 10.4.4 Update #47 to 11.0.9 Update #23
  • Rebooted on almost daily basis
  • All crons stopped for the times corresponding to the server crashes
  • Created a MySQL log to monitor the lock times on queries

Possible causes:

  • Failing hardware
  • Incorrect software configuration (MySQL, Apache etc)

Responsibilities:

  • Small webserver
  • Runs our billing system - WHMCS
  • Responsible for CRONs
  • Bulk-email solution - No delivery times coincide with server crashes

Proposed solutions:

  • Move machine over to VM
  • Format and restore the Plesk server backup and take it from there?

Side notes:

  • Seems to be a general Apache failure across all our linux servers - Intermittent problem
  • Are we doing something fundamentally wrong in the Apache config? (I understand that this is a secondary question, just making sure that it isnt possibly holding any relevance)
share|improve this question
add comment

3 Answers

up vote 2 down vote accepted

I never use prtg, but if I am reading the chart correctly, you are running out of memory. And your server problem last, if not completely crashed, from around 1am to 2-3am. Though the problem seem to start from 12am. Your server load just jump to the roof right at that moment.

During that period of time:

  • Chart Memory (Swap) Free 2, swap usage build up to 6G-7G, that is a lot comparing with 1G of physical ram
  • Chart Memory (Real) Free 2 / SNMP Linux Meminfo 2, all ram are used

Though memory seems to be the main cause. It is possible (or part of the problem) caused by lack of cpu power. As previous request still being process, new request comes in, more and more request pile up in the server.

I would suggest increasing memory, and also find out what is being run at 12am.

share|improve this answer
    
It seems that no crons are run at that time. So we have eliminated the possibility of crons. There are 2 RAM slots still open. will look at upgrading the RAM and see if the problem is resolved –  deanvz Nov 23 '12 at 9:11
    
What about backup? How is storage configure? Is it iscsi/nfs? –  John Siu Nov 24 '12 at 0:28
    
It is isci. The backup runs at 1 AM in the morning. Upgraded the 2 GB of RAM to 4GB. –  deanvz Nov 26 '12 at 10:52
    
Any improvement? How is backup done? (from san/iscsi / from server / from another server) Also is there timezone/day light saving difference between backup config and the server? If they are off by 1hr ... –  John Siu Nov 28 '12 at 3:48
    
There has been an improvement yes. From server to another server, it does have a private/backup network interface. No there isnt (daylight savings). We are in South Africa. –  deanvz Nov 29 '12 at 6:11
add comment

Sounds like you need to do some real analysis of the root cause(s).

  • Configure and monitor apache's server-status to get a feel for web server load.
  • Set up system monitoring for basic metrics (CPU, memory, disk activity) to see where exactly the bottleneck is
  • Monitor dmesg closely, both when you restart and during normal running, to verify there are no obvious hardware issues.

Once you have a few days' worth of solid data, you can take the next step (the one you thought you were taking now - ask for advice.)

share|improve this answer
    
We make use of PRTG to monitor the server. So we looked at these (PRTG) logs in order to find a pattern, but with no luck. We logged the MySQL query times and then tried to match long query times with server load and found no link. What is the best way to monitor dmesg? Run a few terminals with dmesg | grep -i memory etc? –  deanvz Nov 5 '12 at 14:35
    
You're going to have to provide these results, of course. –  adaptr Nov 5 '12 at 14:35
    
postimage.org/image/85q1vqgy9/full - this should give you a broad idea. I can provide any detailed logs if need be –  deanvz Nov 5 '12 at 14:49
    
Ugh, that's far too small to be of any use. –  adaptr Nov 5 '12 at 15:13
    
Can surely zoom in, pic is in high res –  deanvz Nov 6 '12 at 10:45
show 1 more comment

99.9% of the time on a setup like the one you have it's misconfiguration of mysql on a box that is too small to handle the amount of connections assigned. A very average setup of mysql sets the connection limit to 200, each connection that comes in usually takes between 10 ~ 100mb depending on the queries / caching etc.

I've seen many companies setting their connection limits way over the maximum memory that the actual machine has based on how they configure it. When MySQL tries to address the memory and gets assigned to swap instead it causes the system to crash. You can usually see traces in dmesg.

Post your MySQL configuration + number of cpu's/vcpus and memory, likelyhood it's probably MySQL that is configured incorrectly. The documentation is hard to follow for mysql but there are some helper scripts to give you an idea. I'll try to find one of the ones I've used in the past that is the most accurate unfortunately i don't recall the script name off the top of my head.

Also keep in mind that looking at mysql logs will not show you the true story.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.