Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am running Ubuntu 12.04 LTS. Yesterday I found a message in my mailbox saying that my server was shut down. I proceeded to reboot the system, but it didn't come up after many minutes, and I didn't have a hardware KVM system to see what the kernel was printing to the terminal. So I rebooted the system to a Linux rescue image and I saw that the software RAID 1 array was out of sync. The rescue system also began to reconstruct the RAID array.

So far there is no evidence that any of the disks have hardware errors. SMART statuses look good so far.

I never received an email notification by mdadm, even though email notification was turned on in /etc/mdadm/mdadm.conf.

This server was also configured to forward all syslog messages to a log host, so I checked my log host. The relevant parts are:

May 20 15:38:40 kernel: [    1.869825] md0: detected capacity change from 0 to 536858624
May 20 15:38:40 kernel: [    1.870687]  md0: unknown partition table
May 20 15:38:40 kernel: [    1.877412] md: bind
May 20 15:38:40 kernel: [    1.878337] md/raid1:md1: not clean -- starting background reconstruction
May 20 15:38:40 kernel: [    1.878376] md/raid1:md1: active with 2 out of 2 mirrors
May 20 15:38:40 kernel: [    1.878418] md1: detected capacity change from 0 to 3000052808704
May 20 15:38:40 kernel: [    1.878575] md: resync of RAID array md1
[snip]
May 20 15:52:33 kernel: Kernel logging (proc) stopped.
May 20 15:52:33 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="845" x-info="http://www.rsyslog.com"] exiting on signal 15.

As you can see, the system (the normal one, not the rescue system) already detected that something was wrong with the RAID array during a system boot. Then, shortly after, something (not me) halted the system.

So my questions are:

  1. What could cause the disks to suddenly become out of sync?
  2. Why was I not notified by email?
  3. Why was the error not properly logged to syslog before halting the system? Could it be that the system tried to log to syslog, but did so after stopping the syslog daemon? If so what can I do to prevent that?
  4. What can I do to find out what happened? Or, if there's no way for me now to find out what happened, how can I improve logging and notifications so that next time I can do a better post-mortem?

My question is not about proper backup practice. I already know that RAID is not a backup etc. My question is solely about notifications and diagnosis.

share|improve this question

1 Answer 1

What could cause the disks to suddenly become out of sync?

Drive failure, controller failure, some other hardware failure. Some obscure software issue.

Why was I not notified by email?

Ubuntu has a cronjob /etc/cron.d/mdadm that results in the RAID volumes being checked once a day at 00:57. If your system wasn't having problems then, or it had already failed by then, then there was no way to send a message.

Why was the error not properly logged to syslog before halting the system?

Well, if drives are failing it doesn't really make sense to try to write to them since any further writes could trash whatever is left. Not knowing the exact nature of your failure, it could be that your volume or filesystem went read-only. By default Ubuntu is setup to switch to a read-only filesystem if there are errors on the root volume.

how can I improve logging and notifications so that next time I can do a better post-mortem?

Setup logging to a remote syslog host. That way a storage failure doesn't mean that nothing can be logged.

share|improve this answer
    
I already have logging to a remote syslog host, and I still didn't get a message. Why? And why would sending email fail? Sending email doesn't require storage right? –  Hongli Lai May 21 '13 at 21:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.