1

We have been dogged by Server 2003 "locking up" from time to time since we moved from OLD linux servers to the new machines. It appears that the I/O waits go through the roof a few times a day and everyone gets hung until the O/S can get around to clearing it. Overall everyone reports better responsiveness from the new server, except when it gets clogged up with this I/O backlog.

We're unsure of how to solve it though. We only have 16 users hitting this server, and there's only 8GB of space used across its drives (RAID 10, all 15K drives). Performance counters for disk/network/memory/cpu are all close to 0 ... except for the average disk queue length, which shoots up at the same time that users complain.

In Linux, load averages would never go above 2 or 3, while now the queue length will sometimes shoot up to 10 or 12 even though we have faster drives, more drives, more memory, more cores, same exact application hitting the server, etc.

First question - any suggestions for what we can do to track down and resolve this?

Second question - is there a way to create a logical disk drive in memory (then just DFS Replicate it with a physical hard drive or even write scripts to replicate it from our app)? The whole data drive is only 8GB (and the server supports 48GB of memory), but I'm not sure how to handle this.

Improve this question
3
  • Interesting question. Can you describe the IO behavior of the app that's causing the problem - file read\writes or database or other?. How are your disks partitioned? Are your OS\paging\data volumes on separate spindles? How many disks? Have you aligned the data volume? Which version of Windows 2003 32 or 64bit?
    – Helvick
    Feb 9, 2010 at 6:52
  • App is a flat file database (similar to SQLite) with all shared db files residing on the file share. OS/data are on different drives. 4 data drives, RAID 10. 2003 Server 64.
    – Beep beep
    Feb 9, 2010 at 14:26
  • 16 (very busy) users driving at full tilt shouldn't normally saturate a RAID 10 4 drive 15k SAS volume but it's certainly not impossible with a flat file DB. What RAID controller are you using? If file locking is not the root cause and IOPS on the drives is then moving the DB to a SSD's with RAID 1 might be an option but it does seem odd that this sort of use case is a problem on this hardware wasn't an issue on the older Linux server.
    – Helvick
    Feb 9, 2010 at 15:20

2 Answers 2

Reset to default
2

Sounds like a driver issue or hardware failure. I'd start by updating all the storage and chipset related drivers and hardware firmware.

You may also want to contact the manufacturers' support line(s) and ask if they have seen a situation like this. I know as crazy as this sounds, half the time they have a quick solution (or at least can identify the problem).

If you're storage array has a management program check it for errors or drive health information. Check for remapped sectors or other indicators that a drive is having problems.

Improve this answer
3
  • Hardware is where I'd look first. Run drive & array integrity checks from your RAID controller if at all possible (can usually be done non-destructively, but probably requires an outage)
    – voretaq7
    Feb 9, 2010 at 4:03
  • Yep, we did all that. HP said it could just be our app, which hits the drive real hard with updates ... still doesn't make sense to me why it would only happen on the new server, not the old.
    – Beep beep
    Feb 9, 2010 at 4:04
  • I'm still leaning toward hardware issues, but anything is possible. Have you tried hosting the database on a different windows machine temporarily to see if there's any difference? I also question HP's Win2k3 64-bit drivers, that was not a popular OS and it's possible there's few enough users that a bug slipped through.
    – Chris S
    Feb 11, 2010 at 14:10
0

Chris S already gave you great pointers on where to start looking. Re: your second question, I wouldn't advise RAMDrives (even DFS-Replicated ones) for production data you care about.

If your DFS replication is synchronous then your RAM drive will probably lock up if writes to the physical drive you're replicating to stall, and if your replication scheme is not synchronous you will lose power (and your UPS battery will be mysteriously dead) while the physical copy is in an inconsistent (or outdated) state: Instant data loss via Murphy's law.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .