Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a CentOS 6 server which is running CPanel. SugarCRM was installed on a playground domain and nothing further was done with it. I began to notice an unbelievable drop in available space, and overnight I dug around and discovered that sugar had apparently created a cron job which was piping a great deal of output into a hidden log file. I killed the cronjob and removed the (17gb) worth of log files. I didn't realize that CPanel was set up to install updates automatically, and during this process CPanel attempted to install an update for which it didn't have enough space. MySql server crashed as (according to the log file) it didn't have enough space to write the pid or sock file. I have not been able to start it yet. service mysqld restart yields the message: unrecognized service. service mysql restart yields the following messages:

MySQL manager or server PID file could not be found! [FAILED]

Starting MySQL.Manager of pid-file quit without updating fi[FAILED]

There is no mysql pid file in /var/lib/mysql. I have checked permissions:

ls -la /var/lib/ | grep mysql 
drwxr-xr-x 30 mysql mysql 4096 Aug 26 10:50 mysql/

there is nothing in the /var/lock/mysql file, and I don't see any lock files.

server.com.err file does have a message that the table ./mysql/user is marked as crashed and should be repaired, though I've googled for hours and have tried several methods of repairing this and have not yet been successful.

mysqld_safe yields the following messages:
mysqld_safe Logging to '/var/lib/mysql/server.com.err'
mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
mysqld_safe mysqld from pid file /var/lib/mysql/server.com.pid ended

Also, I am completely missing the mysqld directory in /var/lib/mysql. When I try:

/usr/sbin/mysqld --skip-grant &

I get the following:

[ERROR] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!
[ERROR] Aborting
[Note] /usr/sbin/mysqld: Shutdown complete

What am I to do? I've performed a complete (and duplicate) backup of all of the databases found in /var/lib/mysql on a local hard drive and also to an ftp location on a different host. Is the only thing left to do remove and reinstall mysql server? I do have CPanel so I want to make sure what I do is compatible with that.

share|improve this question
3  
I guess I learned a lot from that one. –  user1867350 Aug 26 '13 at 19:52
    
Sorry we didn't get to his answer faster, but always be extremely suspicious of commands that involve the rm utility. At least you can restore from backup. –  Chris S Aug 26 '13 at 21:50
    
agreed. a memorable learning experience to be sure! I did have a snapshot though, which I've since restored from. Would removing and reinstalling mysql have been the way to go otherwise, in this case? –  user1867350 Aug 27 '13 at 3:53

1 Answer 1

up vote 1 down vote accepted

Oh, that looks rather screwed up. Haven't yet had mysql.user corruption, but I can imagine that being troublesome. I think the best bet (besides restoring from an earlier backup) is to set up mysql on another server, recreate the grants you had, shut down that server and copy the /var/lib/mysql/mysql/ directory over to the broken one.

share|improve this answer
    
I almost did something like that actually. I think that going forward I'm going to mirror web and database servers and rather than serving both together from same server I will split them up, utilizing remote mysql access host so I have only one location to update should one or the other fail, minimizing downtime and (if I understand this correctly) precluding the need to modify several scripts as well. Anyway, marked as answered because I think this is definitely what I should have done. –  user1867350 Aug 27 '13 at 4:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.