I am experiencing a weird problem where the CentOS 6.5 system I was using spontaneously underwent a restart that was not authorized by me, and then enters Rescue Mode, where I am unable to diagnose the issue further.
This server is a new server, which is one of the reasons I was initially confused. This happened twice, and the second time, I logged all of my actions in order to see if there was a potential problem I caused.
My process began as follows as logged into the root user:
- Installed EPEL and with it,
- Created user
- Set up an
authorized_keysfile with my RSA pub key and another person's key
- Altered permissions:
chmod 700 /home/deploy/.ssh
chmod 400 /home/deploy/.ssh/authorized_keys
- Changed password for
deployto the sudoers list (
- Changed the following lines in the
iptables -P INPUT ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 21 -j ACCEPT
service iptables save
service iptables restart
As far as I know, everything I had done up to this point was correct procedure. It was as I was proceeding with creating some MySQL databases that I received this output suddenly:
Broadcast message from email@example.com (unknown) at 2:22 ... The system is going down for reboot NOW! Control-Alt-Delete pressed Connection to 22.214.171.124 closed by remote host. Connection to 126.96.36.199 closed.
Not only did this catch me off-guard, but I was trying to think about how this could have possibly happened because at the time, I was the only user who was connected to the server, and I was in the middle of creating a user in the MySQL command prompt as well. I am confident that I did not press CTRL+ALT+DEL on my keyboard.
Proceeding this point, the standard email came from SoYouStart, the server provider, informing me that the server had been put into rescue mode. Upon entering rescue mode, I referred to this guide as a reference for how I might diagnose the issue. I initially found out what my hard drive partitions were with
fdisk -l. This server uses a software RAID, and the bigger RAID partition was on the
Therefore, I tried running
fsck /dev/md3 to try to see if there was an issue with the filesystem to no resolve. I also tried doing a SMART test on the hard drives as well, and they both passed without any significant errors.
Following this, I mounted the drive using
mount /dev/md3 /mnt/md3 and tried going into the filesystem. I tried editing
rc.local file and added this (source):
#!/bin/sh -e sudo iptables -X sudo iptables -t nat -F sudo iptables -t nat -X sudo iptables -t mangle -F sudo iptables -t mangle -X sudo iptables -P INPUT ACCEPT sudo iptables -P FORWARD ACCEPT sudo iptables -P OUTPUT ACCEPT
After this, doing a
reboot just put the system back into Rescue Mode.
I'm at a loss for what could possibly be happening. I'm not sure if it's a user error or something else. I've done this process on numerous other servers, including a different dedicated server with the same provider, but I'm running out of options for what could have caused it.