0

A Debian Server stopped responding earlier today. After hardware reset, the data on server was rolled back several months ago to the state it was in february. How is this possible?Everything, logs, webdata, databases, are as if the computer has been turned off on 03. February and turnd on today.

Does anyone know what could cause this? Hacked? How can I prevent this from happening again?

Thanks a lot!

EDIT:

Here is the update what really happened.

On the Server there is a Mirror RAID configuration. One of the HDDs stopped writing data on February 3rd. When second HDD broke earlier today, the system could only boot from the first one. That is why all the data was that old.

Apparently, this is one in the million situation.

EDIT:

So you don't get into same situation as me:
http://www.tcpdump.com/kb/os/linux/raid-alerting.html

Improve this question
8
  • That's bizzare, no hack could do that. The only thing I can think is that you have been running this in a virtual environment and a snapshot was rolled back. Where is this server situated? Please provide as much info as possible to get a proper answer, in it's current form your answer will not be answered and will most likely be closed.
    – Alex Berry
    Jul 16, 2012 at 11:34
  • Two clarIfying questions. 1) is this a VM? 2) is it mounting storage from elsewhere (NFS or similar)?
    – EEAA
    Jul 16, 2012 at 11:42
  • It is a Hetzner EQ6 Server. (www.hetzner.de). Thanks for the info, I will contact the hosting company.
    – Bojan Hrnkas
    Jul 16, 2012 at 11:51
  • It's much more common than you think. If you are leasing a server with built-in RAID, you should monitor it yourself, and ticket the provider if you ever see the slightest hiccup with the RAID array or an individual drive.
    – Michael Hampton
    Jul 16, 2012 at 14:20
  • @MichaelHampton what kind of monitoring tool would you suggest for debian?
    – Bojan Hrnkas
    Jul 16, 2012 at 16:25

3 Answers 3

Reset to default
2

Here is the update what really happened.

On the Server there is a Mirror RAID configuration. One of the HDDs stopped writing data on February 3rd. When second HDD broke earlier today, the system could only boot from the first one. That is why all the data was that old.

So you don't get into same situation as me: http://www.tcpdump.com/kb/os/linux/raid-alerting.html

Improve this answer
0

if you're using unionfs to overlay changes and that overlay died or was reset, you'd end up "winding the clock back", but that would be quite unusual.

Improve this answer
0

The mdadm package comes with its own scripts to monitor the raid status daily and send an email upon failure:

/etc/cron.daily/mdadm

# cron.daily/mdadm -- daily check that MD devices are functional
(..)
# As recommended by the manpage, run
#      mdadm --monitor --scan --oneshot
# every day to ensure that any degraded MD devices don't go unnoticed.
# Email will go to the address specified in /etc/mdadm/mdadm.conf

Normally the user that gets the emails is configured as root in /etc/mdadm/mdadm.conf so in order to send the emails to a different address, instead of the local root account, add an entry like this to /etc/aliases:

root: [email protected]

Suffice it to say your MTA such as exim should be configure to be able to send out email.

Improve this answer
2
  • There is a more detailed tutorial under the link I provided in my initial post. It also uses mdadm.
    – Bojan Hrnkas
    Aug 14, 2012 at 6:16
  • Yes, but since the debian mdadm package comes with such functionality "out of the box" I figured to mention it. It may save some time.
    – aseq
    Aug 15, 2012 at 4:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .