I am experiencing a weird problem where the CentOS 6.5 system I was using spontaneously underwent a restart that was not authorized by me, and then enters Rescue Mode, where I am unable to diagnose the issue further.
This server is a new server, which is one of the reasons I was initially confused. This happened twice, and the second time, I logged all of my actions in order to see if there was a potential problem I caused.
My process began as follows as logged into the root user:
yum update
- Installed EPEL and with it,
fail2ban
- Created user
deploy
withuseradd
- Set up an
authorized_keys
file with my RSA pub key and another person's key - Altered permissions:
chmod 700 /home/deploy/.ssh
chmod 400 /home/deploy/.ssh/authorized_keys
- Changed password for
root
anddeploy
and addeddeploy
to the sudoers list (visudo
) - Changed the following lines in the
sshd_config
:PermitRootLogin no
PasswordAuthentication no
AllowUsers deploy
- Altered
iptables
configuration:iptables -P INPUT ACCEPT
iptables -F
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 21 -j ACCEPT
service iptables save
service iptables restart
As far as I know, everything I had done up to this point was correct procedure. It was as I was proceeding with creating some MySQL databases that I received this output suddenly:
Broadcast message from [email protected]
(unknown) at 2:22 ...
The system is going down for reboot NOW!
Control-Alt-Delete pressed
Connection to 192.99.10.69 closed by remote host.
Connection to 192.99.10.69 closed.
Not only did this catch me off-guard, but I was trying to think about how this could have possibly happened because at the time, I was the only user who was connected to the server, and I was in the middle of creating a user in the MySQL command prompt as well. I am confident that I did not press CTRL+ALT+DEL on my keyboard.
Proceeding this point, the standard email came from SoYouStart, the server provider, informing me that the server had been put into rescue mode. Upon entering rescue mode, I referred to this guide as a reference for how I might diagnose the issue. I initially found out what my hard drive partitions were with fdisk -l
. This server uses a software RAID, and the bigger RAID partition was on the md3
partition.
Therefore, I tried running fsck /dev/md3
to try to see if there was an issue with the filesystem to no resolve. I also tried doing a SMART test on the hard drives as well, and they both passed without any significant errors.
Following this, I mounted the drive using mount /dev/md3 /mnt/md3
and tried going into the filesystem. I tried editing rc.local
file and added this (source):
#!/bin/sh -e
sudo iptables -X
sudo iptables -t nat -F
sudo iptables -t nat -X
sudo iptables -t mangle -F
sudo iptables -t mangle -X
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT
After this, doing a reboot
just put the system back into Rescue Mode.
I'm at a loss for what could possibly be happening. I'm not sure if it's a user error or something else. I've done this process on numerous other servers, including a different dedicated server with the same provider, but I'm running out of options for what could have caused it.