We have a RHEL5 install on a machine with 5 Seagate 1 TB SAS disks. One has the OS and swap on it. The other four are in a hardware Raid 5 (mounted as /home) managed by a DELL Perc 6/i controller. When the system boots up it spends a long time on udev, which eventually times out. Nonetheless the write speed on the raid is in 90-100MB/s range. We tested write speed by copying a large file (~3 GB) over and over. On the seventh copy the write speed suddenly fell to 1 MB/s. We have tested this multiple times and the problem is reproducible. There are no messages in any of the logs in /var/log/. free -m and vmstat do not show any swap activity. A reply to a similar post asked for contents of /proc/mdstat. Well ours are:
# cat /proc/mdstat Personalities : unused devices: <none>
I am not sure how to trace the problem beyond this. This system used to work fine for a over a year and half. This problem started after we tried to upgrade from 1 TB disks to 2 TB disks to increase disk space. The new disks were installed and array built from scratch. We have since reverted to the 1 TB disks since we know that hardware config worked for sure. Any suggestions or troubleshooting tips are welcome. Thank you for your time and patience.
EDIT: Problem solved. Turns out that the machine needed many firmware updates. Around the time the 2 TB disks were first tried the RAID controller was updated, and this was causing issues. Now after installing 3 BIOS and 1 newer Controller firmware updates the machine works like a charm. Write speeds are in the 180 MB/s range. Thanks to the people who tried to help.