0

An EC2 instance running Apache 2.2.5 on CentOS 6.4 appeared to be fine up until September 29. That day at 7:49 the Value in CloudWatch was 42, at 7:59 it was in the low 100,000's, the last Value report was at 8:09 in the low 600's and then it appears to have completely went down from a stats perspective.

This is the apache log from that day:

Sep 29 08:14:27 ip-172-31-46-50 kernel: type=1305 audit(1411978467.613:5654223): audit_pid=0 old=797 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
Sep 29 08:14:27 ip-172-31-46-50 kernel: type=1305 audit(1411978467.714:5654224): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditctl_t:s0 res=1
Sep 29 08:14:27 ip-172-31-46-50 kernel: Kernel logging (proc) stopped.
Sep 29 08:14:27 ip-172-31-46-50 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="813" x-info="http://www.rsyslog.com"] exiting on signal 15.

telnet ip 80 returns a refused connection; nmap -p 80 ip show 80/tcp as closed

Port 80 is open in the Security Group: EC2 Console screenshot

Second issue, maybe related, is that mysql will only start with mysqld_safe. Trying to start mysql via service mysqld start fails and the message is:

141003 20:33:49 InnoDB: Using Linux native AIO
141003 20:33:49 InnoDB: Initializing buffer pool, size = 128.0M
141003 20:33:49 InnoDB: Completed initialization of buffer pool
141003 20:33:49  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
InnoDB: File name ./ibdata1
InnoDB: File operation call: 'open'.
InnoDB: Cannot continue operation.

The user in my.cnf is mysql; This is the data directory as specified in my.cnf.

-rwx------. 1 mysql mysql 748683264 Oct  3 20:19 ibdata1
-rwx------. 1 mysql mysql   5242880 Oct  3 20:19 ib_logfile0
-rwx------. 1 mysql mysql   5242880 Oct  3 20:19 ib_logfile1
drwx------. 2 mysql mysql      4096 Oct  7  2013 mysql
drwx------. 2 mysql mysql      4096 Oct  7  2013 performance_schema

It's been suggested that apparmor could be the culprit; however, I've ran apparmor_status (command not found) and did a find / -name "*apparmor*" to no avail - I'll check further on this though.

Through mysqld_safe I was able to get a dump so I now have everything backed up, but would really like to resolve as opposed to a new instance.

Thanks for any help!

4
  • 1. Is apache running? 2. If so, can you curl localhost? 3. When you restart apache, are there any relevant entries in error_log or access_log? 4. Are you sure you are running apparmor? The syslog looks like selinux. if so, check if you are enforcing or permissive. If you are enforcing, try changing it to permissive, reboot and see if it fixes the issue.
    – pete.k
    Oct 4, 2014 at 2:21
  • Apache is running, I can curl http://localhost:80 (logged in via ssh as root), I can not curl http://domain.com (from local terminal) as I get curl: (7) Failed connect to domain.com:80; Connection refused (replaced actual domain). It is not apparmor, you're correct. I changed to permissive and was then able to start mysql; however, upon reboot it was set back and I could not start mysql again. Oct 6, 2014 at 14:18
  • Further: stopping iptables, setting to permissive, and starting mysql allows access to the app. Oct 6, 2014 at 14:39
  • I have mysqld working correctly now. There is a non standard datadir and (possibly during a reboot) the directories were relabeled and didn't have the correct context. I updated the contexts and set SELinux back to enforcing, all appears to be well with mysqld now. Oct 6, 2014 at 15:35

0

You must log in to answer this question.

Browse other questions tagged .