We have 7x 4-core quad-cpu (i.e. 16 vCPU's); 7x 2-core quad-cpu (i.e. 8 vCPU's); 4x dual-core (i.e. 2 CPU's) All running Citrix Presentation Server for Windows v4.5 on Windows Server 2003
Citrix clients (and any admin RDP session) experience typical "freezes" in session interactivty referred to in this and many other forums. These coincide with disk-queue overruns and dropped counters.
These servers are all out-of-the-box configured ... no tuning has been applied.
Each blade has two blade-mounted HDD's in RAID 1 (mirror) config.
What we see happen is the PhysicalDisk\Avg. Disk Queue Length on all these servers frequently extends well into the 40's! The more vCPU's the more often, longer duration and greater the length of the queues. This is always directly related to the WRITE queue length ... almost entirely UNrelated to any READ activity.
I have identified that this is not caused by any ONE application or process but seems to happen when several processes all throw write activities at the system at once and sustain that level of write activity for more than a few seconds.
The lazywrter appears to get demanded and works frantically eventually tapering off as it appears to catch-up with writing stuff to disk, but the odd thing is that there is never a high THROUGHPUT of I/O during these queue extensions. What I mean is that at times I've seen the disks burst to write speeds of upto 15 MBytes/s but during these events, they rarely exceed 2 MBytes/s. During these events the Memory\Cache Bytes can often double (e.g 300Mb to nearly 600MB)
In all the forums these SYMPTOMS are discussed over and over but nowhere can I find what CAUSES this nor what to DO to fix this.
We have hardware caching enabled but not yet done any registry tweaks so plan to first apply the HKLM\System\CurrentControlSet\Control\Session Manager\Configuration Manager\RegistryLazyFlushInterval tweak which was supposed to go hand-in-glove with enabling caching.
I would love to understand why this is happening (and what can be done about it) because these boxes are NOT heavily loaded either with users (max 12 typ 5~6) nor in terms of Network (7~30%) or CPU (<40% _total) activity and even when these evenst occur it's not like millions of IOPs get thrown at the system drive ... more like thousands, and it seems like as soon as the cache goes over about 300 MB thats when the Lazywriter kicks up and the disk queues go mad.
Anyone insight to the cause, would be appreciated.