I have reports that I/O on a certain filesystem, namely /srv/data1, is intermittently horrific. I've seen it myself and it's true. For example, I run 'ls' /srv/data1 and the output takes 30+ seconds to appear. I run it again and this time, no delay. The periods of OK and horrific are short and cycle rather rapidly.
/srv/data1 is a 500GB OCFS2 on /dev/drbd1, which in turn is on systemA:/dev/md3 and systemB:/dev/md3. A and B are identical Dell R610s running SLES11. On both A and B /dev/md3 is a mirror of /dev/sdc1 and /dev/sdd1. And those are 500 GB SATA drives used for nothing else. The HDD controller is Dell's PERC 6/iR.
This is a busy, busy master Subversion server and the delays are breaking some processes that keep remote proxy servers in sync with this master. That's causing trouble for the development teams.