We're having a problem with data inconsistencies on an IBM server we've set up, containing 6 1TB SAS drives running of an IBM raid controller in RAID 6. I would subscribe it to faulty drive(s), only the RAID controller isn't reporting any problems. The file systems (all ext3) have been remounted in read-only mode by the OS (Open Suse 11) several times, and the server then has to be restarted, fsck'ed, and then booted again. Any thoughts as to what could be wrong?
Firmware for the controller and the drives should be set to latest or most stable (you can ask the IBM techsupport for the version numbers and download links)
also, ext3 is limited in file sizes as well as partition size. check out http://en.wikipedia.org/wiki/Ext3 to make sure you're not overusing the FS.
Also, LSI based controllers used to be limited to 2Tb logical disk size, that was resolved with recent firmware. it is possible that you are using an early firmware which is not quite stable for large logical disk sizes.
RAID 6 engines are fairly new, could be a firmware bug. Hardware engineers are not the always the best software developers.
I would troubleshoot by first building a JBOD. Then expand to RAID 5, 6. If it turns out to be the hardware RAID engine, you could go with software RAID.
Personally, I don't like RAID 3+ at all. Compared to RAID 10 you get more space or more availability for the price of a 4x performance hit on small writes and dodgy recovery on drive failure.
Are you using an LSI SAS controller?
We had an issue on those, that any SMART activity (tests, status etc.) causes the controller to reset. This sets the disks faulty and degrades RAID.
Try disabling stuff like smartctl, hddtemp etc. Basicly disable anything that uses S.M.A.R.T.
Could be the server quietly corrupting the raid set. Possible if for example there was a faulty bus. Frequently syslog / dmesg would make mention of APIC and or Interupts problems around the time of failure.