Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

How I can rebuild raid arrays? I am using Raid 1. My datacentar says it needs to be fixed, first i thought it is HDDs faulty because of smartmoontools scanning result but it is not.

command:

cat /proc/mdstat

output:

Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[1] sda1[0]
      2096064 blocks [2/2] [UU]

md1 : active raid1 sda2[0]
      524224 blocks [2/1] [U_]

md2 : active raid1 sda3[0]
      729952192 blocks [2/1] [U_]

unused devices: <none>

Does I need to:

# mdadm /dev/md1 -r /dev/sdb2
# mdadm /dev/md2 -r /dev/sdb3
# mdadm /dev/md3 -r /dev/sdb4

and then

# mdadm /dev/md1 -a /dev/sdb2
# mdadm /dev/md2 -a /dev/sdb3
# mdadm /dev/md3 -a /dev/sdb4 

Will I lose data or my server will be offline?

Here is the output for fdisk -l

Disk /dev/sda: 750.1 GB, 750156374016 bytes
64 heads, 32 sectors/track, 715404 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               2        2048     2096128   fd  Linux raid autodetect
/dev/sda2            2049        2560      524288   fd  Linux raid autodetect
/dev/sda3            2561      715404   729952256   fd  Linux raid autodetect

Disk /dev/sdb: 750.1 GB, 750156374016 bytes
64 heads, 32 sectors/track, 715404 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               2        2048     2096128   fd  Linux raid autodetect
/dev/sdb2            2049        2560      524288   fd  Linux raid autodetect
/dev/sdb3            2561      715404   729952256   fd  Linux raid autodetect

Disk /dev/md2: 747.4 GB, 747471044608 bytes
2 heads, 4 sectors/track, 182488048 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md2 doesn't contain a valid partition table

Disk /dev/md1: 536 MB, 536805376 bytes
2 heads, 4 sectors/track, 131056 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/md0: 2146 MB, 2146369536 bytes
2 heads, 4 sectors/track, 524016 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md0 doesn't contain a valid partition table

Here is output for smartctl -A /dev/sdb

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   111   100   006    Pre-fail  Always       -       38042073
  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       7
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   073   060   030    Pre-fail  Always       -       24494887
  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       7935
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       7
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       4
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   062   052   045    Old_age   Always       -       38 (Min/Max 34/41)
194 Temperature_Celsius     0x0022   038   048   000    Old_age   Always       -       38 (0 26 0 0 0)
195 Hardware_ECC_Recovered  0x001a   032   026   000    Old_age   Always       -       38042073
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       101494372179726
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       3317006641
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2924590852
share|improve this question
1  
I don't see a /dev/md3 in your cat /proc/mdstat. You won't have to remove partitions from the RAID when they aren't there, you only have to use the -a commands. The output of fdisk -l would also help. –  etagenklo Apr 7 '13 at 12:47
    
I added output to post. –  Luka Apr 7 '13 at 12:51
    
If I do this sig-io.nl/?p=210 I will not loose any data? –  Luka Jun 1 '13 at 22:40

3 Answers 3

up vote 4 down vote accepted
+50

That drive sdb looks like it's not far from failing. Though it hasn't officially failed yet, it doesn't have much life left in it.

195 Hardware_ECC_Recovered  0x001a   032   026   000    Old_age   Always       -       38042073

This drive has had a large number of recoverable read errors. Which means it successfully reconstructed the data using error correction. However, it's getting to the point where it is most likely it will soon have an unrecoverable read error, where it cannot successfully reconstruct data on a damaged or failing section of the disk. At that point there's nothing you can do and you'll have to replace the drive.

If your rebuild keeps stopping, at the same place, it's entirely possible the drive has already failed at that point on the platters, and isn't reporting it. Desktop class drives will stop and try for minutes or even hours to read a particular sector if they fail the first time, which leads to this sort of thing. And you probably have such a drive in this "server"...

At this point you should have that drive proactively replaced, since it is going to fail soon, if not already.

share|improve this answer
    
OK, so, I have two Hard drives... /dev/sdb and /dev/sda, /dev/sdb needs to be replaced? If datacenter replace /dev/sdb I need to copy data from /dev/sda to /dev/sdb? –  Luka Jun 1 '13 at 23:13
    
Once you replace the drive, you'll have to tell mdadm to use it to restore your RAID1 mirrors onto it. The Linux RAID wiki has lots of information you can use, including tutorials on recovery. –  Michael Hampton Jun 1 '13 at 23:23
    
Is the /dev/sda ORIGINAL or /dev/sdb? :( I do not need to backup any data if I tell datacenter to replace /dev/sdb? I just need to rebuild raid arrays to new disk? –  Luka Jun 1 '13 at 23:27
    
They're mirrors, so "original" has no meaning. And yes, you always need to backup, whether you have failing disks or not (but that's another discussion). –  Michael Hampton Jun 1 '13 at 23:30
    
I have backup on ext. server of course, but I really do not want to install cpanel and set up everything again, it has a lot of settings, optimization etc etc I am asking you is /dev/sdb faulty and when they replace it, will i boot my system as normal after rebuilding raid with new disk in rescue mode? –  Luka Jun 1 '13 at 23:30

It' seems that only one half of the mirror is missing. So there should be no problem.But the question is why are the sumbirrors (sdbX) missing? Maybe it would be a good idea to check them before reatachment to the mirrors.

mdadm --manage /dev/md1 --add /dev/sdb2
mdadm --manage /dev/md2 --add /dev/sdb3
mdadm --manage /dev/md3 --add /dev/sdb4
share|improve this answer
    
Thank you. It is recovering md2 now :) This is good? md2 : active raid1 sdb3[2] sda3[0] 729952192 blocks [2/1] [U_] [>....................] recovery = 0.0% (5120/729952192) > finish=16377.0min speed=26297K/sec –  Luka Apr 7 '13 at 12:53
1  
It's syncing really slow. 731K/sec. If the recovery stays as slow, please copy the output of smartctl -A /dev/sdb in your question. –  etagenklo Apr 7 '13 at 12:55
    
No 26297K/sec :) I edited now :) I am watching it watch cat /proc/mdstat –  Luka Apr 7 '13 at 12:56
1  
looks better. I'd still recommend to check smart values, because there usually is a reason for partitions to get kicked out of the RAID. –  etagenklo Apr 7 '13 at 12:58
    
Datacenter did HDDs test with 30 minutes offline, and they say there is not faulty. I wil check again with smartmoontools when syncing is finished :) thank you guys a lot –  Luka Apr 7 '13 at 12:59

When you have two volumes mirrored using RAID1, each copy has an "activity counter" that gets updated when something gets changed inside the copy: that way the system is able to understand which one is the most up to date in case of a crash or other things that take down one of the two.

Resyncing means that the most current gets copied over the older one, which "lost sync". So an offline hard drive or a defective one will force the array into the "degraded mode" (just one copy online, non redundancy).

From a degraded mode you can recover either forcing a resync so that the only active partition gets cloned to the one which you have taken back to work, or replacing the defective disk and providing the new space to the array, which will cloned in the same way described before.

Both methods will keep your data intact, unless you do something unproper to configurations or to the "good" partitions which are still alive =)

I guess it is a good idea to backup your raid configurations before you even think to fiddle with them =)

As to the SMART values, they seem quite ok to me, besides the Hardware_ECC_Recovered which is also discussed on other replies..

Anyway avoid considering the single value you see there, also check at which pace it is changing. I once had strange values on a drive, but they didn't get worst, they were stable. On the other hand good SMART values are not proof of a perfect disk: in my opinion they can be good to check disk wearing due to usage/aging, but they can do little to prevent sudden failures (for instance those caused by mechanical stress, overheating, etc.. - think of a failing cooler on the server room..)

Good luck! =)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.