/etc/mdadm/mdadm.conf
ARRAY /dev/md0 level=raid5 num-devices=4 metadata=00.90 UUID=54e55d36:c3971e21:d484c166:6e86c3d7
devices=/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde
MAILADDR root
mdadm -D /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Thu Apr 30 23:48:58 2009
Raid Level : raid5
Array Size : 4395415488 (4191.79 GiB 4500.91 GB)
Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Tue Jul 5 10:21:41 2011
State : clean, degraded
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
UUID : 54e55d36:c3971e21:d484c166:6e86c3d7
Events : 0.28886
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
3 0 0 3 removed
after each reboot array is degraded and /dev/sde is removed. I re-add /dev/sde using
mdadm /dev/md0 -a /dev/sde
array getting rebuilt everything seems OK.
Until the next reboot...
EDIT
dmesg
1.275758] ata3.00: ATA-8: ST31500341AS, CC1H, max UDMA/133
[ 1.275764] ata3.00: 2930277168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[ 1.275793] ata4.00: ATA-8: ST31500341AS, CC1H, max UDMA/133
[ 1.275799] ata4.00: 2930277168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[ 1.296314] ata1.00: ATAPI: HL-DT-STDVD-ROM GDR8164B, 0L06, max UDMA/33
[ 1.296714] ata1.01: HPA detected: current 156299375, native 156301488
[ 1.296720] ata1.01: ATA-6: ST380011A, 3.06, max UDMA/100
[ 1.296725] ata1.01: 156299375 sectors, multi 16: LBA48
[ 1.303263] ata3.01: ATA-8: ST31500341AS, CC1H, max UDMA/133
[ 1.303267] ata3.01: 2930277168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[ 1.303375] ata4.01: ATA-8: ST31500341AS, CC1H, max UDMA/133
[ 1.303378] ata4.01: 2930277168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[ 1.308845] ata1.00: configured for UDMA/33
[ 1.320424] ata1.01: configured for UDMA/100
[ 1.324633] scsi 0:0:0:0: CD-ROM HL-DT-ST DVD-ROM GDR8164B 0L06 PQ: 0 ANSI: 5
[ 1.324914] scsi 0:0:1:0: Direct-Access ATA ST380011A 3.06 PQ: 0 ANSI: 5
[ 1.359697] ata4.00: configured for UDMA/133
[ 1.367662] ata3.00: configured for UDMA/133
[ 1.415713] ata4.01: configured for UDMA/133
[ 1.424171] ata3.01: configured for UDMA/133
[ 1.424284] scsi 2:0:0:0: Direct-Access ATA ST31500341AS CC1H PQ: 0 ANSI: 5
[ 1.424515] scsi 2:0:1:0: Direct-Access ATA ST31500341AS CC1H PQ: 0 ANSI: 5
[ 1.424753] scsi 3:0:0:0: Direct-Access ATA ST31500341AS CC1H PQ: 0 ANSI: 5
[ 1.424971] scsi 3:0:1:0: Direct-Access ATA ST31500341AS CC1H PQ: 0 ANSI: 5
[ 1.447254] sr0: scsi3-mmc drive: 52x/52x cd/rw xa/form2 cdda tray
[ 1.447260] Uniform CD-ROM driver Revision: 3.20
[ 1.447418] sr 0:0:0:0: Attached scsi CD-ROM sr0
[ 1.448228] sd 0:0:1:0: [sda] 156299375 512-byte logical blocks: (80.0 GB/74.5 GiB)
[ 1.448622] sd 0:0:1:0: [sda] Write Protect is off
[ 1.448626] sd 0:0:1:0: [sda] Mode Sense: 00 3a 00 00
[ 1.448656] sd 0:0:1:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1.448848] sda:
[ 1.449175] sd 2:0:0:0: [sdb] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB)
[ 1.449246] sd 2:0:0:0: [sdb] Write Protect is off
[ 1.449250] sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 1.449281] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1.449478] sdb:
[ 1.449759] sd 3:0:0:0: [sdd] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB)
[ 1.449818] sd 3:0:0:0: [sdd] Write Protect is off
[ 1.449821] sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00
[ 1.449847] sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1.450030] sdd:
[ 1.453309] sr 0:0:0:0: Attached scsi generic sg0 type 5
[ 1.453369] sd 0:0:1:0: Attached scsi generic sg1 type 0
[ 1.453420] sd 2:0:0:0: Attached scsi generic sg2 type 0
[ 1.453476] sd 2:0:1:0: Attached scsi generic sg3 type 0
[ 1.453532] sd 3:0:0:0: Attached scsi generic sg4 type 0
[ 1.454032] sd 3:0:1:0: Attached scsi generic sg5 type 0
[ 1.465468] sda1 sda2 <
[ 1.478018] sd 3:0:1:0: [sde] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB)
[ 1.478399] sd 2:0:1:0: [sdc] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB)
[ 1.478609] unknown partition table
[ 1.478667] sd 2:0:1:0: [sdc] Write Protect is off
[ 1.478670] sd 2:0:1:0: [sdc] Mode Sense: 00 3a 00 00
[ 1.478700] unknown partition table
[ 1.478819] sd 2:0:1:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1.479134] sd 3:0:0:0: [sdd] Attached SCSI disk
[ 1.479149] sdc:
[ 1.479284] sd 3:0:1:0: [sde] Write Protect is off
[ 1.479287] sd 3:0:1:0: [sde] Mode Sense: 00 3a 00 00
[ 1.479317] sd 3:0:1:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1.479498] sde: sda5 >
[ 1.495522] sd 0:0:1:0: [sda] Attached SCSI disk
[ 1.500601] sd 2:0:0:0: [sdb] Attached SCSI disk
[ 1.500604] unknown partition table
[ 1.500852] sd 2:0:1:0: [sdc] Attached SCSI disk
[ 1.508325] unknown partition table
[ 1.509078] sd 3:0:1:0: [sde] Attached SCSI disk
[ 1.877343] async_tx: api initialized (async)
[ 1.878206] xor: automatically using best checksumming function: pIII_sse
[ 1.896005] pIII_sse : 6515.000 MB/sec
[ 1.896009] xor: using function: pIII_sse (6515.000 MB/sec)
[ 1.964021] raid6: int32x1 735 MB/s
[ 2.032073] raid6: int32x2 753 MB/s
[ 2.100050] raid6: int32x4 550 MB/s
[ 2.168011] raid6: int32x8 490 MB/s
[ 2.236012] raid6: mmxx1 2367 MB/s
[ 2.304019] raid6: mmxx2 2598 MB/s
[ 2.372010] raid6: sse1x1 1589 MB/s
[ 2.440024] raid6: sse1x2 1978 MB/s
[ 2.508009] raid6: sse2x1 2976 MB/s
[ 2.576004] raid6: sse2x2 3575 MB/s
[ 2.576006] raid6: using algorithm sse2x2 (3575 MB/s)
[ 2.587036] md: raid6 personality registered for level 6
[ 2.587040] md: raid5 personality registered for level 5
[ 2.587042] md: raid4 personality registered for level 4
[ 2.619394] md: md0 stopped.
[ 2.622452] md: bind<sdc>
[ 2.624170] md: bind<sdd>
[ 2.625842] md: bind<sdb>
[ 2.632060] raid5: device sdb operational as raid disk 0
[ 2.632065] raid5: device sdd operational as raid disk 2
[ 2.632068] raid5: device sdc operational as raid disk 1
[ 2.632566] raid5: allocated 4222kB for md0
[ 2.632751] 0: w=1 pa=0 pr=4 m=1 a=2 r=4 op1=0 op2=0
[ 2.632755] 2: w=2 pa=0 pr=4 m=1 a=2 r=4 op1=0 op2=0
[ 2.632758] 1: w=3 pa=0 pr=4 m=1 a=2 r=4 op1=0 op2=0
[ 2.632761] raid5: raid level 5 set md0 active with 3 out of 4 devices, algorithm 2
[ 2.632806] RAID5 conf printout:
[ 2.632808] --- rd:4 wd:3
[ 2.632811] disk 0, o:1, dev:sdb
[ 2.632813] disk 1, o:1, dev:sdc
[ 2.632815] disk 2, o:1, dev:sdd
[ 2.632852] md0: detected capacity change from 0 to 4500905459712
[ 2.635265] md0: unknown partition table
[ 2.692382] PM: Starting manual resume from disk
[ 2.692387] PM: Resume from partition 8:5
[ 2.692389] PM: Checking hibernation image.
[ 2.700369] PM: Error -22 checking image file
[ 2.700373] PM: Resume from disk failed.
[ 2.752886] kjournald starting. Commit interval 5 seconds
[ 2.752902] EXT3-fs: mounted filesystem with ordered data mode.
[ 4.475170] udev[370]: starting version 164
[ 5.095175] input: PC Speaker as /devices/platform/pcspkr/input/input1
[ 5.096407] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input2
[ 5.096415] ACPI: Power Button [PWRB]
[ 5.097714] ACPI: SSDT 1f5e74c0 0026C (v01 PmRef Cpu0Ist 00003000 INTL 20040311)
[ 5.100680] processor LNXCPU:00: registered as cooling_device0
[ 5.101730] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3
[ 5.101742] ACPI: Power Button [PWRF]
[ 5.105041] ACPI: SSDT 1f5e7980 00152 (v01 PmRef Cpu1Ist 00003000 INTL 20040311)
[ 5.107531] processor LNXCPU:01: registered as cooling_device1
[ 5.264144] parport_pc 00:08: reported by Plug and Play ACPI
[ 5.264369] parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
[ 6.020497] i801_smbus 0000:00:1f.3: PCI INT B -> GSI 19 (level, low) -> IRQ 19
[ 6.021314] intel_rng: FWH not detected
[ 6.113334] [drm] Initialized drm 1.1.0 20060810
[ 6.228124] pci 0000:00:02.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 6.228132] pci 0000:00:02.0: setting latency timer to 64
[ 6.232413] pci 0000:00:02.0: irq 27 for MSI/MSI-X
[ 6.232469] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
[ 6.241459] Error: Driver 'pcspkr' is already registered, aborting...
[ 6.255289] HDA Intel 0000:00:1b.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 6.255345] HDA Intel 0000:00:1b.0: setting latency timer to 64
[ 6.338649] input: HDA Digital PCBeep as /devices/pci0000:00/0000:00:1b.0/input/input4
[ 6.989723] Adding 2650684k swap on /dev/sda5. Priority:-1 extents:1 across:2650684k
[ 7.259580] EXT3 FS on sda1, internal journal
[ 7.500343] loop: module loaded
[ 7.537631] it87: Found IT8718F chip at 0x290, revision 5
[ 7.537642] it87: in3 is VCC (+5V)
[ 7.558886] coretemp coretemp.0: Using relative temperature scale!
[ 7.558938] coretemp coretemp.1: Using relative temperature scale!
[ 8.122878] kjournald starting. Commit interval 5 seconds
[ 8.128716] EXT3 FS on md0, internal journal
[ 8.128724] EXT3-fs: mounted filesystem with ordered data mode.
[ 8.898492] r8169: eth0: link up
[ 8.898497] r8169: eth0: link up
[ 16.029883] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 16.116193] nf_conntrack version 0.5.0 (7879 buckets, 31516 max)
[ 16.116698] CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
[ 16.116702] nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
[ 16.116704] sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
[ 19.080019] eth0: no IPv6 routers present
[ 169.429012] md: bind<sde>
[ 169.483527] RAID5 conf printout:
[ 169.483532] --- rd:4 wd:3
[ 169.483537] disk 0, o:1, dev:sdb
[ 169.483540] disk 1, o:1, dev:sdc
[ 169.483544] disk 2, o:1, dev:sdd
[ 169.483547] disk 3, o:1, dev:sde
[ 169.483622] md: recovery of RAID array md0
[ 169.483625] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 169.483634] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 169.483639] md: using 128k window, over a total of 1465138496 blocks.
EDIT-2
mdadm --assemble -v --force /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 4.
mdadm: added /dev/sdc to /dev/md0 as 1
mdadm: added /dev/sdd to /dev/md0 as 2
mdadm: no uptodate device for slot 3 of /dev/md0
mdadm: added /dev/sde to /dev/md0 as 4
mdadm: added /dev/sdb to /dev/md0 as 0
mdadm: /dev/md0 has been started with 3 drives (out of 4) and 1 spare.
