4

I'm having a strange issue that I've never encountered before. I'm running a Turnkey Linux LAMP server (Debian) and it seems that my MySQL server becomes inaccessible at least once a day. I'm not sure what's causing it at all. My last few logs before I restart it are:

160108  0:54:09 [Note] Plugin 'FEDERATED' is disabled.
160108  0:54:09 InnoDB: The InnoDB memory heap is disabled
160108  0:54:09 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160108  0:54:09 InnoDB: Compressed tables use zlib 1.2.8
160108  0:54:09 InnoDB: Using Linux native AIO
160108  0:54:09 InnoDB: Initializing buffer pool, size = 128.0M
160108  0:54:09 InnoDB: Completed initialization of buffer pool
160108  0:54:09 InnoDB: highest supported file format is Barracuda.
160108  0:54:09  InnoDB: Waiting for the background threads to start
160108  0:54:10 InnoDB: 5.5.46 started; log sequence number 111777334
160108  0:54:11 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.46-0+deb8u1'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  (Debian)

I haven't changed any of the default settings that I can recall so it should be listening on 3306. I have a few Wordpress sites running on the server so having the DB go down on a whim is sort of bad news. It comes right back up when I restart with no problems and says it's listening on 3306:

160108 10:20:45 [Note] Server socket created on IP: '127.0.0.1'.
160108 10:20:45 [Note] Event Scheduler: Loaded 0 events
160108 10:20:45 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.46-0+deb8u1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)

Any ideas? Thanks!

UPDATE: Here is my full log file: http://pastebin.com/2G2CAVsw

THE PROBLEM: It appears that tklbam-restore causing the problem. I manually ran a backup and noticed that as soon as it got to the DB phase of the process, my Wordpress servers could no longer access MySQL. Also, the backup process seems to be stuck at one of my DB tables. Here's the last few lines:

table: trendsandteens/wp_wfNet404s
table: trendsandteens/wp_wfReverseCache
table: trendsandteens/wp_wfScanners
table: trendsandteens/wp_wfStatus
table: trendsandteens/wp_wfThrottleLog
table: trendsandteens/wp_wfVulnScanners

It's just backing up Wordfence's tables. So I'm not really sure what the issue is... Any ideas? Here is the traceback after I interrupted the process: http://pastebin.com/QV63cBPG

15
  • run the command dmesg and see if you see anything that OOM has killed mysql
    – Mike
    Jan 8, 2016 at 15:33
  • More of the MySQL log would be helpful. And you can prevent WP from being inaccessible with webserver level caching, or solutions like Varnish or CloudFront.
    – JayMcTee
    Jan 8, 2016 at 15:44
  • @Mike I'm not seeing anything from OOM in dmesg or anything about MySQL.
    – anguiac7
    Jan 8, 2016 at 15:46
  • @JayMcTee I'll probably go for Varnish since that will handle the whole server. To my knowledge, you have to individually add sites to CloudFront and you can't just add an entire server, correct?
    – anguiac7
    Jan 8, 2016 at 15:46
  • 1
    @SPRBRN I'm running the TunkeyLinux LAMP distro (turnkeylinux.org/lampstack) which includes tklbam (turnkeylinux.org/docs/tklbam). Maybe I'll ask around on their forums. I'm not sure how active the TK community is on serverfault.
    – anguiac7
    Jan 9, 2016 at 20:29

1 Answer 1

1

Try starting MySQL using strace and store the output to a file. Then review the output right before it terminates to see if there's anything which would point to the cause of the issue.

Be warned though that the output can grow quite large, so make sure you don't run out of disk space or otherwise negatively impact the system (such as if it requires a lot of I/O to write all the data to the disk).

If you find that strings were cut off that would be helpful to have to investigate further, use the -s argument to strace.

If it's easier, you can attach strace to an existing process using -p processid.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .