up vote 0 down vote favorite
1
share [g+] share [fb]

We have serious stability issues with mysqld running on Linux hosts in EC2, with all of its data and log files stored on an EBS volume. We keep a slave purely for hot backup and failover, and when the master goes down, we can usually bring up the slave as a master without any issues, and then create a new slave.

But it's very problematic that our master will just go down. The master host keeps running fine, but mysqld won't respond to anything, and can't even be killed with kill -9.

This happens in both our production and staging environments, which are similar, but production runs on large instances (with Centos 5.2 x86_64) and staging on medium instances (with Centos 5.2 i686).

Has anybody experienced similar mysqld stability problems in EC2, and if so, how did they deal with them?

Thanks in advance.

link|improve this question
25% accept rate
1  
When you say "mysqld won't respond to anything", what sorts of things (other than kill -9 ;) have you tried? – Ben Dunlap Aug 15 '09 at 17:31
No mysql clients can connect -- not the web application, not mysql, not mysqladmin. They just freeze and themselves can't be stopped with ^C or ^Z. – user5336 Aug 17 '09 at 14:01
feedback

3 Answers

up vote 3 down vote

If mysqld won't die even with a kill -9, then the problem is almost certainly that it's in uninterruptible sleep waiting for disk IO. This strongly suggests that you've got a dud EBS, which happens sometimes. If you're feeling excessively optimistic, you can try contacting Amazon support, but the quickest solution is just to create a new EBS and use that (hopefully you'll be on a less-crap storage unit) or try moving to a different availability zone. Yes, they're bollocks options, but EC2 just glitches up like that sometimes and you're effectively screwed.

link|improve this answer
Followup: RightScale, whom we use to intermediate our EC2 instances, has told us about a new kernel image that resolves a bug whereby xfs_freeze would fail and block reads/writes to the EBS storage. Here's to hoping it works, so that I don't have to do something like moving to the volatile EC2 filesystem and backing up to S3.... Many thanks for your insights! – user5336 Aug 17 '09 at 18:25
Changing the kernel image appears to have solved the problem, thankfully. xfs_freeze + old kernel image didn't play together nicely. – user5336 Sep 21 '09 at 3:07
feedback
up vote 0 down vote

Agreed. We have some long running ec2mysql instances and have had no issues. It sounds like a hardware issue specific to your environment.

link|improve this answer
feedback
up vote 0 down vote

Try to connect as root (ie the mysql root user, not your normal root user). It's possible that there are too many connections to mysql, which prevents new connections. The mysql root account is except from these restrictions and can always connect.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.