MySQL Crashing and failing to restart

Question

MySQL seems to be crashing, normally later in the day but I can't see why! It says out of memory but there is loads of free ram and no swap space being used. I'm using Azure.

Here is the log:

Oct  3 20:42:20 GenyxLive kernel: [787828.711240] Out of memory: Kill process 53891 (mysqld) score 84 or sacrifice child
Oct  3 20:42:20 GenyxLive kernel: [787828.714081] Killed process 53891 (mysqld) total-vm:871780kB, anon-rss:58164kB, file-rss:0kB
Oct  3 20:42:20 GenyxLive kernel: [787828.731974] init: mysql main process (53891) killed by KILL signal
Oct  3 20:42:20 GenyxLive kernel: [787828.733525] init: mysql main process ended, respawning
Oct  3 20:42:20 GenyxLive kernel: [787828.974636] init: mysql main process (24975) terminated with status 1
Oct  3 20:42:20 GenyxLive kernel: [787828.974666] init: mysql main process ended, respawning
Oct  3 20:42:21 GenyxLive kernel: [787829.937186] init: mysql post-start process (24976) terminated with status 1
Oct  3 20:42:22 GenyxLive kernel: [787830.103158] init: mysql main process (25002) terminated with status 1
Oct  3 20:42:22 GenyxLive kernel: [787830.103194] init: mysql respawning too fast, stopped

Here's the graph of whats happening:

You can see a spike of cpu and then the mysql service stops...

Can anyone help me figure out whats happening?

Ubuntu 12.04

Azure Small (768mb ram, 1ghz cpu)

EDIT

There is a cron job that runs every 5 mins which just checks for new domains etc (using zpanel).

Its a 64 bit OS

Linux GenyxLive 3.2.0-48-virtual #74-Ubuntu SMP Thu Jun 6 20:02:55 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

free -lm result:

             total       used       free     shared    buffers     cached
Mem:           672        491        180          0         42        118
Low:           672        491        180
High:            0          0          0
-/+ buffers/cache:        330        341
Swap:            0          0          0

cat /proc/meminfo result:

MemTotal:         688348 kB
MemFree:          186788 kB
Buffers:           43956 kB
Cached:           120988 kB
SwapCached:            0 kB
Active:           335908 kB
Inactive:          84924 kB
Active(anon):     255976 kB
Inactive(anon):      296 kB
Active(file):      79932 kB
Inactive(file):    84628 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                16 kB
Writeback:             0 kB
AnonPages:        255924 kB
Mapped:            18568 kB
Shmem:               376 kB
Slab:              51788 kB
SReclaimable:      37052 kB
SUnreclaim:        14736 kB
KernelStack:        1344 kB
PageTables:         7632 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      344172 kB
Committed_AS:     967220 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       13192 kB
VmallocChunk:   34359723128 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       51136 kB
DirectMap2M:      735232 kB

Answer 1

Kernel OOM only kicks in when it considers resources are really tight... so you must be running out of memory but the chart doesn't make much sense then.

What does free -lm show? Also, what does cat /proc/meminfo show?

In terms of a trigger for the issue, does some maintenance process or other cron job run around this time which might be doing queries where mysql needs to maintain temp tables in memory?

BTW... is this a 32-bit version of the OS? Just wondering if your low memory is exhausted.

EDIT 1 Ok. Well, mysqld will be the single most memory consuming process so that is most likely why it is being picked on. Check your apache config to see how may worker threads you are spawning... each will have a memory overhead and it is possible that when more spawn you run out of memory as you have no swap space. Check your MaxClients parameter... if you have any more than 50 (just a ballpark figure) this might be a good place to look. Might I recommend taking a look at Apache Performance Tuning.