Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

A Debian Server stopped responding earlier today. After hardware reset, the data on server was rolled back several months ago to the state it was in february. How is this possible?Everything, logs, webdata, databases, are as if the computer has been turned off on 03. February and turnd on today.

Does anyone know what could cause this? Hacked? How can I prevent this from happening again?

Thanks a lot!

EDIT:

Here is the update what really happened.

On the Server there is a Mirror RAID configuration. One of the HDDs stopped writing data on February 3rd. When second HDD broke earlier today, the system could only boot from the first one. That is why all the data was that old.

Apparently, this is one in the million situation.

EDIT:

So you don't get into same situation as me:
http://www.tcpdump.com/kb/os/linux/raid-alerting.html

share|improve this question
    
That's bizzare, no hack could do that. The only thing I can think is that you have been running this in a virtual environment and a snapshot was rolled back. Where is this server situated? Please provide as much info as possible to get a proper answer, in it's current form your answer will not be answered and will most likely be closed. –  Alex Berry Jul 16 '12 at 11:34
    
Two clarIfying questions. 1) is this a VM? 2) is it mounting storage from elsewhere (NFS or similar)? –  EEAA Jul 16 '12 at 11:42
    
It is a Hetzner EQ6 Server. (www.hetzner.de). Thanks for the info, I will contact the hosting company. –  Bojan Hrnkas Jul 16 '12 at 11:51
    
It's much more common than you think. If you are leasing a server with built-in RAID, you should monitor it yourself, and ticket the provider if you ever see the slightest hiccup with the RAID array or an individual drive. –  Michael Hampton Jul 16 '12 at 14:20
    
@MichaelHampton what kind of monitoring tool would you suggest for debian? –  Bojan Hrnkas Jul 16 '12 at 16:25

3 Answers 3

up vote 2 down vote accepted

Here is the update what really happened.

On the Server there is a Mirror RAID configuration. One of the HDDs stopped writing data on February 3rd. When second HDD broke earlier today, the system could only boot from the first one. That is why all the data was that old.

So you don't get into same situation as me: http://www.tcpdump.com/kb/os/linux/raid-alerting.html

share|improve this answer

if you're using unionfs to overlay changes and that overlay died or was reset, you'd end up "winding the clock back", but that would be quite unusual.

share|improve this answer

The mdadm package comes with its own scripts to monitor the raid status daily and send an email upon failure:

/etc/cron.daily/mdadm

# cron.daily/mdadm -- daily check that MD devices are functional
(..)
# As recommended by the manpage, run
#      mdadm --monitor --scan --oneshot
# every day to ensure that any degraded MD devices don't go unnoticed.
# Email will go to the address specified in /etc/mdadm/mdadm.conf

Normally the user that gets the emails is configured as root in /etc/mdadm/mdadm.conf so in order to send the emails to a different address, instead of the local root account, add an entry like this to /etc/aliases:

root: name@example.org

Suffice it to say your MTA such as exim should be configure to be able to send out email.

share|improve this answer
    
There is a more detailed tutorial under the link I provided in my initial post. It also uses mdadm. –  Bojan Hrnkas Aug 14 '12 at 6:16
    
Yes, but since the debian mdadm package comes with such functionality "out of the box" I figured to mention it. It may save some time. –  aseq Aug 15 '12 at 4:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.