5

We have a 6 SSD, RAID5 volume. The SSD disks we are using are Crucial M4-CT512M4SSD2 firmware 000F. We write 100 ~ 200GB data through MySQL InnoDB to the volume daily. The RAID5 volume is only used for MySQL and nothing else.

Output of lsscsi:

    [0:0:0:0]    disk    iDRAC    LCDRIVE          0323  /dev/sdb 
    [1:0:0:0]    cd/dvd  iDRAC    Virtual CD       0323  /dev/sr0 
    [1:0:0:1]    disk    iDRAC    Virtual Floppy   0323  /dev/sdc 
    [2:2:0:0]    disk    DELL     PERC H700        2.10  /dev/sda 
    [3:0:0:0]    disk    ATA      M4-CT512M4SSD2   000F  /dev/sdd 
    [3:0:1:0]    disk    ATA      M4-CT512M4SSD2   000F  /dev/sde 
    [3:0:2:0]    disk    ATA      M4-CT512M4SSD2   000F  /dev/sdf 
    [3:0:3:0]    disk    ATA      M4-CT512M4SSD2   000F  /dev/sdg 
    [3:0:4:0]    disk    ATA      M4-CT512M4SSD2   000F  /dev/sdh 
    [3:0:5:0]    disk    ATA      M4-CT512M4SSD2   000F  /dev/sdi 
    [3:0:6:0]    enclosu LSI CORP SAS2X36          0717  -       
    [3:0:7:0]    enclosu LSI CORP SAS2X36          0717  -       

Output of smartctl : (see here for attribute 173)

for p in d e f g h i ; do smartctl -A /dev/sd${p} | grep 173  ; done

173 Unknown_Attribute       0x0033   098   098   010    Pre-fail  Always       -       67
173 Unknown_Attribute       0x0033   098   098   010    Pre-fail  Always       -       66
173 Unknown_Attribute       0x0033   098   098   010    Pre-fail  Always       -       71
173 Unknown_Attribute       0x0033   098   098   010    Pre-fail  Always       -       66
173 Unknown_Attribute       0x0033   098   098   010    Pre-fail  Always       -       69
173 Unknown_Attribute       0x0033   097   097   010    Pre-fail  Always       -       105

And the output for /proc/mdstat:

Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdi[6] sdg[3] sdh[4] sdf[2] sdd[0] sde[1]
  2500532160 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
  bitmap: 3/4 pages [12KB], 65536KB chunk

 unused devices: <none>

We are unsure why "sdi" is more than 40% more worn out than other disks, as the parity should be evenly distributed across all disks.

2
  • Exactly how long have these drives been in production? Sep 14, 2012 at 7:59
  • sorry for the very late reply. The drives have been running since July 2012
    – Alex Zheng
    Oct 27, 2012 at 12:18

2 Answers 2

4

It's just rounding error. This is the smallest difference SMART is capable of showing. Likely over the next few days, the other SSDs will switch to 097 one by one.

7
  • Sorry, I was talking about the last column from smartctl output. The last column "RAW_VALUE" for 5 devices for attribute 173 is around 70. But for "sdi" it is over 100. According to the manual for this SSD (micron.com/~/media/Documents/Products/Technical%20Note/…) the value for 173 is "Average erase count of all good blocks" The values you are talking about is value 202 "Percentage Of The Rated Lifetime Used" thanks
    – Alex Zheng
    Sep 5, 2012 at 1:25
  • You're still talking about a microscopic difference, much less than one third of one percent. Sep 5, 2012 at 1:43
  • Thanks, I agree with you one device is currently at 97% life and the rest is at 98%. What I am asking is the raw value for attribute 173, which for my setup says for "sdi", the average erase count for the flash chip is 105 cycles, while for the others is only 66 ~ 71. This is more than 40% difference in the average erase count.
    – Alex Zheng
    Sep 5, 2012 at 2:04
  • @AlexZheng: You are measuring in a senseless way. It's like looking at the cars at an auto dealer and not wanting the one that has driven "8 times as far as the others" because it went 16 miles and the others went 2. You have to look at this as a fraction of the life of the drive. By that measure, it's much less than one percent -- a microscopic difference. Similarly, the cars in the dealership all have way less than one percent of their life used, so you can consider them all the same. Check when they have measurable use and they'll converge. Sep 5, 2012 at 2:20
  • Thanks, and again, I agree with you. These SSD drive are rated for more than 3,000 erase cycles. In that regard it is a very small difference. But it does not explain why this one drive is worn out 40% more than the other drives, as part of the same RAID, created at the same instance of time. These drives have not been individually benchmarked in any way, so our workload is the only reason they can be worn out. Also, if this pattern continues, that one SSD will hit the rated 3,000 erase cycle while the other drives were only at 2000. Even though that is not for a few years later.
    – Alex Zheng
    Sep 5, 2012 at 2:41
0

RAID5 distributes parity across the disks, this is true, but your FS can be not that evenly spread across it. Specially when it comes to EXT{2,3,4} with their old-school superblocks schemas. With EXT{2,3,4} FS, being improperly aligned, you can have superblocks updates going to one disk more often than to others. Check it out: https://raid.wiki.kernel.org/index.php/RAID_setup#Calculation

XFS does it automatically. ;-)

Also, you can easily get actual disk statistics with iostat, so any anomalies in regards of disks load would be clearly displayed.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .