1

I'm a web developer and I am having a very nasty problem with one of the websites I'm working on and I was hoping somebody here may be able to help me.

The website is running on a dedicated server with CentOS 6.6, an Nginx + Apache configuration with Vesta as control panel. I'm not sure if it's very relevant here, but the website uses Sphinx as search engine.

Since mid January every 6 days the server crashes, every time at different hours. The recovery usually takes around an hour and 15 minutes time during which there are no entries in any of the log files.

After the server recovery, 2 of the largest tables in the MySQL database throw duplicate key errors and because they are too large to repair quickly enough I usually truncate them and then restore them from backups.

  • I checked all the logs and I could not find any hints/relevant information about the crash. All the logs contain normal entries until the time of the crash and then resume after the server recovery.

  • I checked all the crons for all the users and there is none that runs every 6 days.

  • the CPU and memory usage before the crashes is very low: 1.6% CPU and 16.5% memory, which is the usual load on the server.

  • for about a week I suspected that the Vesta backup cron was somehow related to this, as it was keeping the memory usage at 74% at all times, even after it finished running, but I disabled it and apart from the decrease of memory usage, the crash is still there.

Do you have any advice on what I can do to identify the culprit? I've ran out of ideas.

Thanks!

PS: If you need me to provide other information, please let me know!

5

1 Answer 1

0

Try to collect metrics and graph them. Nothing beats graphs. A tools like Munin can be very helpful in these situations look at memory, io, processes, cpu, networking, interrupts, etc. over time.

http://munin-monitoring.org/

Also if your machine is a VM and has a network filesystem that becomes unavailable, that may explain the gap in the log times (for extra points try logging remotely).

1
  • Thanks for the suggestion! So far I've been checking the CPU, memory and disk usages with sar. VestaCP has some graphs too, but they don't provide much detail.
    – Fallen
    Mar 11, 2015 at 11:24

Not the answer you're looking for? Browse other questions tagged .