Using Linux mdadm, I've setup a six disk raid10,f2 array. I'm using 7200RPM SATA drives. The biggest load I am worried about is recording four streams of live video (worst case of 20Mbits each, final file size ends up being 10GB in the worst case), while watching another previously-saved file (similar characteristics).
I've been seeing glitches in the recordings that appear to be dropped frames. I suspect my the RAID array isn't keeping up, but I'd like to know more about what specifically is going on. Is it seeking too much? Is there some random wait? Is there a specific device in the array that's taking longer than the others, etc.
What sort of tools should I be using, what information should I be gathering, and how do I interpret it? Ideally, I'd like to have something gather data for several days then I could go back and look at it, but if that's not possible more live-snapshot type tools will work too.