0

I am experiencing a weird problem where the CentOS 6.5 system I was using spontaneously underwent a restart that was not authorized by me, and then enters Rescue Mode, where I am unable to diagnose the issue further.

This server is a new server, which is one of the reasons I was initially confused. This happened twice, and the second time, I logged all of my actions in order to see if there was a potential problem I caused.

My process began as follows as logged into the root user:

  1. yum update
  2. Installed EPEL and with it, fail2ban
  3. Created user deploy with useradd
  4. Set up an authorized_keys file with my RSA pub key and another person's key
  5. Altered permissions:
    • chmod 700 /home/deploy/.ssh
    • chmod 400 /home/deploy/.ssh/authorized_keys
  6. Changed password for root and deploy and added deploy to the sudoers list (visudo)
  7. Changed the following lines in the sshd_config:
    • PermitRootLogin no
    • PasswordAuthentication no
    • AllowUsers deploy
  8. Altered iptables configuration:
    • iptables -P INPUT ACCEPT
    • iptables -F
    • iptables -A INPUT -i lo -j ACCEPT
    • iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
    • iptables -A INPUT -p tcp --dport 22 -j ACCEPT
    • iptables -P INPUT DROP
    • iptables -P FORWARD DROP
    • iptables -P OUTPUT ACCEPT
    • iptables -A INPUT -p tcp --dport 80 -j ACCEPT
    • iptables -A INPUT -p tcp --dport 21 -j ACCEPT
    • service iptables save
    • service iptables restart

As far as I know, everything I had done up to this point was correct procedure. It was as I was proceeding with creating some MySQL databases that I received this output suddenly:

Broadcast message from [email protected]
    (unknown) at 2:22 ...

The system is going down for reboot NOW!
Control-Alt-Delete pressed 
Connection to 192.99.10.69 closed by remote host.
Connection to 192.99.10.69 closed.

Not only did this catch me off-guard, but I was trying to think about how this could have possibly happened because at the time, I was the only user who was connected to the server, and I was in the middle of creating a user in the MySQL command prompt as well. I am confident that I did not press CTRL+ALT+DEL on my keyboard.

Proceeding this point, the standard email came from SoYouStart, the server provider, informing me that the server had been put into rescue mode. Upon entering rescue mode, I referred to this guide as a reference for how I might diagnose the issue. I initially found out what my hard drive partitions were with fdisk -l. This server uses a software RAID, and the bigger RAID partition was on the md3 partition.

Therefore, I tried running fsck /dev/md3 to try to see if there was an issue with the filesystem to no resolve. I also tried doing a SMART test on the hard drives as well, and they both passed without any significant errors.

Following this, I mounted the drive using mount /dev/md3 /mnt/md3 and tried going into the filesystem. I tried editing rc.local file and added this (source):

#!/bin/sh -e
sudo iptables -X
sudo iptables -t nat -F
sudo iptables -t nat -X
sudo iptables -t mangle -F
sudo iptables -t mangle -X
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

After this, doing a reboot just put the system back into Rescue Mode.

I'm at a loss for what could possibly be happening. I'm not sure if it's a user error or something else. I've done this process on numerous other servers, including a different dedicated server with the same provider, but I'm running out of options for what could have caused it.

1 Answer 1

1

I assume its your server provider in charge. They seems to be sending acpi event to your server to reboot. It could be their reaction on disabling ssh password authorization, ssh key changes or password changes, which they detected with delay.

1
  • I'm tempted to think if it's because I may have accidentally blocked the host's servers in the firewall. I know the host uses other servers and ports to ping the servers on their network, but I may have to contact them to get those addresses to add them to the firewall.
    – J.W.F.
    Jun 16, 2014 at 12:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .