linux/centos crashing periodically

Question

Excuse me if I'm not doing something right on the site it's my first time here (have used stackoverflow before). I have a centos 5.2 server running Plesk and the tech support guys are refusing to look into this as they keep saying that I should just reboot, however that's not really a solution as it's happening a lot (perhaps once a week).

I have to admit I'm not hugely into Linux but I know my way around and here's what the syslem messages say for just before the server went down, can someone perhaps confirm my suspicion that this is a lack of RAM thing?

Dec  7 20:56:24 ns2435 kernel:  [<c10d67f7>] ? handle_mm_fault+0xf7/0x190
Dec  7 20:57:08 ns2435 kernel:  [<c10598e4>] ? do_page_fault+0x164/0x3f0
Dec  7 20:57:26 ns2435 kernel:  [<c10f2757>] ? filp_close+0x47/0x70
Dec  7 20:57:38 ns2435 kernel:  [<c1037df0>] ? do_device_not_available+0x0/0x20
Dec  7 20:57:44 ns2435 xinetd[4186]: START: smtp pid=10839 from=86.11.75.60
Dec  7 20:57:56 ns2435 kernel:  [<c1037dc4>] ? math_state_restore+0x24/0x50
Dec  7 21:02:31 ns2435 kernel:  [<c1059780>] ? do_page_fault+0x0/0x3f0
Dec  7 21:02:41 ns2435 kernel:  [<c19b42ba>] ? error_code+0x5a/0x60
Dec  7 21:02:42 ns2435 kernel:  [<c1059780>] ? do_page_fault+0x0/0x3f0
Dec  7 21:02:45 ns2435 kernel: Mem-Info:
Dec  7 21:02:45 ns2435 kernel: DMA per-cpu:
Dec  7 21:02:45 ns2435 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec  7 21:02:49 ns2435 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Dec  7 21:02:59 ns2435 kernel: Normal per-cpu:
Dec  7 21:03:08 ns2435 kernel: CPU    0: hi:  186, btch:  31 usd: 102
Dec  7 21:03:10 ns2435 kernel: CPU    1: hi:  186, btch:  31 usd:  84
Dec  7 21:02:58 ns2435 xinetd[4186]: START: smtp pid=10862 from=66.220.144.146
Dec  7 21:27:21 ns2435 kernel: active_anon:55616 inactive_anon:55680 isolated_anon:192
Dec  7 21:27:26 ns2435 xinetd[4186]: START: smtps pid=10924 from=86.11.75.60
Dec  7 21:27:29 ns2435 kernel:  active_file:216 inactive_file:399 isolated_file:0
Dec  7 21:27:31 ns2435 xinetd[10862]: warning: can't get client address: Connection reset by peer
Dec  7 21:27:32 ns2435 xinetd[4186]: START: submission pid=10928 from=86.11.75.60
Dec  7 21:27:32 ns2435 kernel:  unevictable:0 dirty:0 writeback:155 unstable:0
Dec  7 21:27:35 ns2435 xinetd[4186]: START: smtp pid=10931 from=86.11.75.60
Dec  7 21:27:35 ns2435 kernel:  free:1349 slab_reclaimable:965 slab_unreclaimable:4086
Dec  7 21:27:38 ns2435 xinetd[4186]: START: smtps pid=10932 from=86.11.75.60
Dec  7 21:27:38 ns2435 kernel:  mapped:221 shmem:25 pagetables:2663 bounce:0
Dec  7 21:27:40 ns2435 kernel: DMA free:2052kB min:88kB low:108kB high:132kB active_anon:6076kB inactive_anon:6308kB active_file:20kB inactive_file:252kB un$
Dec  7 21:27:43 ns2435 kernel: lowmem_reserve[]: 0 484 484 484
Dec  7 21:27:43 ns2435 kernel: Normal free:3344kB min:2768kB low:3460kB high:4152kB active_anon:216388kB inactive_anon:216412kB active_file:844kB inactive_f$
Dec  7 21:27:40 ns2435 xinetd[4186]: START: submission pid=10933 from=86.11.75.60
Dec  7 21:27:56 ns2435 kernel: lowmem_reserve[]: 0 0 0 0

Answer 1

You should try updating to CentOS 5.7 if at all possible, but here are some pointers to help you diagnose the problem:

Look at your memory usage with 'top', see how much memory is available, and how much is used by caches and buffers (can be considered as 'unused' for the sake of simplicity).

You could try monitoring your memory usage over time with something like cacti. Then when you system crashes again, you can see what the situation was before the crash.

If you are running low on available memory, this could be a cause of crashes. If this isn't the case, it could be a problem with bad memory or other hardware issues.

Try installing memtest86 and running that.

Answer 2

You need to issue following commands to analysis what happened on December 07

sar -f /var/logs/sa/sa07  

will provide you information about all day report of cpu usage on december 07

sar -n -f /var/logs/sa/sa07

will provide you information about all day report of network usage on december 07

sar -r -f /var/logs/sa/sa07  

will provide you information about all day report of memory usage on december 07

and if you like check other day of month like 17 the day of month , you need to issue following commands:

sar --f /var/logs/sa/sa17
sar -n -f /var/logs/sa/sa17
sar -b -f /var/logs/sa/sa17
sar -r -f /var/logs/sa/sa17

Answer 3

This kernel log does seem to point towards memory exhaustion. If the server doesn't have the amount of ram that Plesk suggests for recommended operation, then you might be running into ram contention issues with the rest of the system. This could be exacerbated if the server runs multiple logical services.

Answer 4

Do not rule out hardware failure. Take the machine offline for a few hours and run Memtest86. I had a machine once that exhibited consistent failures when I would run a particular application, without fail crash, but it was just one app. On a whim I decided to run Memtest86 and within seconds I had thousands of errors. Pulled the offending memory and I've never had issues since.

Also take note of when the failures occur. For instance how long after reboot do they happen, was the system under heavy load? What was the thermal characteristics like where your server was?

Answer 5

Don't rule in hardware failure before upgrading the kernel. far more likely to be the known bug with MM and that version of the kernel than your ram chips/cosmic rays.

Google "handle_mm_fault" and your kernel version.

Read up on memory handling - the output of free is misleading if you don't understand that linux tends to use all the memory .

Google "linux ate my ram"