An EC2 instance running Apache 2.2.5 on CentOS 6.4 appeared to be fine up until September 29. That day at 7:49 the Value in CloudWatch was 42, at 7:59 it was in the low 100,000's, the last Value report was at 8:09 in the low 600's and then it appears to have completely went down from a stats perspective.
This is the apache log from that day:
Sep 29 08:14:27 ip-172-31-46-50 kernel: type=1305 audit(1411978467.613:5654223): audit_pid=0 old=797 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
Sep 29 08:14:27 ip-172-31-46-50 kernel: type=1305 audit(1411978467.714:5654224): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditctl_t:s0 res=1
Sep 29 08:14:27 ip-172-31-46-50 kernel: Kernel logging (proc) stopped.
Sep 29 08:14:27 ip-172-31-46-50 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="813" x-info="http://www.rsyslog.com"] exiting on signal 15.
telnet ip 80
returns a refused connection; nmap -p 80 ip
show 80/tcp as closed
Port 80 is open in the Security Group:
Second issue, maybe related, is that mysql will only start with mysqld_safe
. Trying to start mysql via service mysqld start
fails and the message is:
141003 20:33:49 InnoDB: Using Linux native AIO
141003 20:33:49 InnoDB: Initializing buffer pool, size = 128.0M
141003 20:33:49 InnoDB: Completed initialization of buffer pool
141003 20:33:49 InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
InnoDB: File name ./ibdata1
InnoDB: File operation call: 'open'.
InnoDB: Cannot continue operation.
The user in my.cnf is mysql
; This is the data directory as specified in my.cnf.
-rwx------. 1 mysql mysql 748683264 Oct 3 20:19 ibdata1
-rwx------. 1 mysql mysql 5242880 Oct 3 20:19 ib_logfile0
-rwx------. 1 mysql mysql 5242880 Oct 3 20:19 ib_logfile1
drwx------. 2 mysql mysql 4096 Oct 7 2013 mysql
drwx------. 2 mysql mysql 4096 Oct 7 2013 performance_schema
It's been suggested that apparmor could be the culprit; however, I've ran apparmor_status
(command not found) and did a find / -name "*apparmor*"
to no avail - I'll check further on this though.
Through mysqld_safe
I was able to get a dump so I now have everything backed up, but would really like to resolve as opposed to a new instance.
Thanks for any help!
curl http://localhost:80
(logged in via ssh as root), I can notcurl http://domain.com
(from local terminal) as I getcurl: (7) Failed connect to domain.com:80; Connection refused
(replaced actual domain). It is not apparmor, you're correct. I changed to permissive and was then able to start mysql; however, upon reboot it was set back and I could not start mysql again.