0

My redhat server is crashing every three weeks or so at 4:15am ish on Sunday mornings. (well it was sundays the last two have been Thursday mornings at 4:15ish) Looking at the logs (mysql, httpd, messages) there are no clues as to why. They just seem to stop.

I ran a little script to take memory readings every 15 minutes and it too stops (with normal readings) at this time.

The server is remote at a provider so I can only access it via the web. I use Plesk.

It appears to be a set job or something that is causing the issue. I can see nothing in crontab.

So my question is...has anyone else had this and can offer advice? Failing that.

Does any one know of a way to get more detailed logging than that offered by the messages file? I was thinking of a black box style recording program or maybe something as simple as an option somewhere to increase the level of reporting in the messages log.

Thanks

Improve this question
1
  • 1
    The classic story of the cleaning crew unplugging a server to plug in a vacuum cleaner comes to mind.
    – Dennis Williamson
    Jun 3, 2010 at 14:08

3 Answers 3

Reset to default
0

this is the times when cron.daily jobs are scheduled, so i would look in /etc/cron.daily or weekly or monthly as the first suspects

you can install atop which will record snapshot of the processes every 10 minutes

alternatively you can install psacct and use accton and lastcomm to see what was being executed

turning on auditing is also an option, see auditd(8)

Improve this answer
3
  • I've installed atop and it looks like just the ticket. Just need to wait for the crash now.
    – columbo
    Jun 4, 2010 at 8:39
  • Thanks Aleksandar. ATOP is proving very useful. I can see my CPU usage rocketing up to 90%ish every day after 4am right about the same time as a mass of awstats.pl processes start. I'm going to try turning off awstats tonight. Thanks.
    – columbo
    Jun 9, 2010 at 14:58
  • PS Anyone who's reading ATOP is really useful.
    – columbo
    Jun 9, 2010 at 14:59
0

You can enable core dumps, which will copy the system memory to a file when a server crashes.

The next problem is what do with the coredump file... If you have someone around who knows all about gdb they may be able to do something about it... or you might be able to use the "strings" command to dump all of the text out of the coredump file, and maybe you'll be able to find something.

Improve this answer
0

log on a different box that is well connected, run screen, ssh into the server and tail kern.log, daemon.log, syslog, messages on individual screen windows. (control-A, c to create a new window, Control-A,D to detach, screen -r to resume)

when the server locks up again, you should have the tail end of the logs in your screen session, even if they haven't been flushed to disk properly when the machine hangs.

If you suspect a kernel panic or oops

kernel.panic=5 kernel.panic_on_oops=5

in your sysctl.conf or equivalent file will wait 5 seconds, possibly allowing the drives to flush, and then will reboot.

Improve this answer
1
  • I don't seem to have kern.log or daemon.log. I have a feeling that reporting on this computer is at a low setting.
    – columbo
    Jun 4, 2010 at 7:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .