Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We're a small shop, running a Dell T420 (dual CPU, only one present, 6 cores) w/32GB RAM as our main server. We have only 5 VMs, one of which is our WSE 2012 DC.

From time to time, and at a rate for which we've not been able to establish a reliable pattern, all of our VMs concurrently spike to 100% CPU. The host remains quiet at 4-5%. A host warm boot doesn't provide relief, but a cold boot at least puts things back in the box until the problem reoccurs.

Sometimes we can get a week or more of calm seas out of it; sometimes only a day. An unreliable pattern seems to be that it kicks off sometime during an extended idle period, i.e. overnight. An examination of the server's temperature logs first led us to suspect overheating, but further investigation into recent incidents have spoiled that lead.

We also found descriptions of similar problems on the Dell forums, with claims of resolution by installing the latest round of Dell updates. We recently engaged in a project to do just that (as an aside, it was quite an adventure getting ~700GB of VHDs safely off of and then back onto that machine), but to our utter dismay it didn't help.

We're absolutely befuddled. So is Microsoft support (or at least first tier support is, even though they try not to act like it). I'm including below our SystemInfo output.

Does anyone know where to start looking?

Thanks

===================================

Host Name:                 SERVER1
OS Name:                   Microsoft Hyper-V Server 2012 R2
OS Version:                6.3.9600 N/A Build 9600
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Server
OS Build Type:             Multiprocessor Free
Registered Owner:          Windows User
Registered Organization:   
Product ID:                06401-029-0000043-76293
Original Install Date:     4/3/2014, 4:07:15 PM
System Boot Time:          5/4/2014, 1:56:47 PM
System Manufacturer:       Dell Inc.
System Model:              PowerEdge T420
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 45 Stepping 7 GenuineIntel ~2200 Mhz
                           [Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20 GHz] (manually added)
BIOS Version:              Dell Inc. 2.1.2, 1/20/2014
Windows Directory:         C:\Windows
System Directory:          C:\Windows\system32
Boot Device:               \Device\HarddiskVolume1
System Locale:             en-us;English (United States)
Input Locale:              en-us;English (United States)
Time Zone:                 (UTC-09:00) Alaska
Total Physical Memory:     32,723 MB
Available Physical Memory: 12,716 MB
Virtual Memory: Max Size:  37,587 MB
Virtual Memory: Available: 17,129 MB
Virtual Memory: In Use:    20,458 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    OIT
Logon Server:              \\SERVER1
Hotfix(s):                 31 Hotfix(s) Installed.
                           [01]: KB2843630
                           [02]: KB2862152
                           [03]: KB2868626
                           [04]: KB2876331
                           [05]: KB2883200
                           [06]: KB2884846
                           [07]: KB2887595
                           [08]: KB2892074
                           [09]: KB2893294
                           [10]: KB2894179
                           [11]: KB2898514
                           [12]: KB2898871
                           [13]: KB2901101
                           [14]: KB2901128
                           [15]: KB2903939
                           [16]: KB2904266
                           [17]: KB2908174
                           [18]: KB2909210
                           [19]: KB2911106
                           [20]: KB2913760
                           [21]: KB2916036
                           [22]: KB2917929
                           [23]: KB2919394
                           [24]: KB2919442
                           [25]: KB2922229
                           [26]: KB2923300
                           [27]: KB2923768
                           [28]: KB2928193
                           [29]: KB2928680
                           [30]: KB2930275
                           [31]: KB2939087
Network Card(s):           3 NIC(s) Installed.
                           [01]: Broadcom NetXtreme Gigabit Ethernet
                                 Connection Name: NIC1
                                 DHCP Enabled:    No
                                 IP address(es)
                           [02]: Broadcom NetXtreme Gigabit Ethernet
                                 Connection Name: NIC2
                                 DHCP Enabled:    Yes
                                 DHCP Server:     192.168.1.12
                                 IP address(es)
                                 [01]: 192.168.1.135
                                 [02]: fe80::915b:8de0:712e:29f1
                           [03]: Hyper-V Virtual Ethernet Adapter
                                 Connection Name: vEthernet (External NIC 1_Internal)
                                 DHCP Enabled:    No
                                 IP address(es)
                                 [01]: 192.168.1.11
                                 [02]: fe80::2d35:f582:4958:9eb2
Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.

share|improve this question

closed as too broad by Andrew B, cole, Grant, mdpc, MichelZ May 8 at 7:01

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

    
Would the reader who down-voted my question come out of hiding and explain why he did so? I tried very hard to provide a thorough explanation of the problem and to demonstrate details of our attempts to resolve it. A foundational tenet of American jurisprudence is the right to confront one's accusers; as an American I greatly value this right and I expect to be allowed to exercise it. Granted I'm new on this forum, but at this moment I'll boldly state that a down-vote without accompanying constructive criticism--especially against a newcomer--is cowardly. –  InteXX May 8 at 0:05
    
Despite the effort placed into troubleshooting the matter, this question falls into the too broad category. As presented, this is a wild goose chase that could very likely be a malware infection. Even that is a shot in the dark. Regardless of the cause, we can't guide you through the entire process of isolating this. If you can narrow things down a bit, there are plenty of people who would be happy to help you. –  Andrew B May 8 at 0:24
    
Regarding the downvote (which I did not leave), we can be overzealous in our voting due to the sheer number of minimal effort do my research for me questions that we get daily. You're a well-intentioned newcomer who at least put some effort in, so this probably warrants being put on hold for being "too broad" more than a downvote. –  Andrew B May 8 at 0:26
    
@AndrewB - Thank you, Andrew, for your candor (as well as for the reversing up-vote, if that was you). I had no idea. Also, if I knew how to narrow things down a bit, I would be absolutely pleased to do so. Alas, as it is I'm a reluctant SysAdmin who feels lucky to know even this little bit that he does. It isn't much farther into the woods than this that I get lost. –  InteXX May 8 at 1:40
    
One thing you can try doing is ruling out malware. It's not fool proof, but try deploying a brand new VM with no network interfaces. (generally this is extreme, but your VM peers are suspect) Leave it running alongside the others. If the CPU does not spike along with the others, then you have at the very least isolated this to traits of the previously deployed VMs. –  Andrew B May 8 at 1:46

1 Answer 1

up vote 1 down vote accepted

It's unclear as to what the problem is; you already know that. We have no chance of telling you what the cause is.

However, you can run some tests:

  • Build VM 1

    • Run a CPU intensive task on this VM constantly
      (Perform millions of complex mathematical calculations per second)
  • Build VM 2

    • Run a RAM intensive task on this VM constantly
      (Create a giant array in memory, delete it, repeat)
  • Build VM 3

    • Run a DISK intensive task on this VM constantly
      (Read/write/delete millions of lines to/from a file)
  • Build VM 4

    • Run a NETWORK intensive task on this VM constantly
      (Copy files to/from a SMB share)

Wait until the problem occurs again, observe performance data on each of these servers.
Which was most affected?
Were any not affected at all?

My guess is that your disks suck and the CPU is waiting for IO operations to complete before continuing, which can cause some applications to flatline the CPU.

share|improve this answer
    
This sounds like quite a project. But I'm up to it. In my secret life I write software (don't tell anyone), so this shouldn't be too tough. Thanks. FWIW, the 8 2TB WD server-grade disks are running a single RAID 6 array. If that tells you anything. So you suspect a race condition? –  InteXX May 8 at 4:21
    
Ah, so that is the term for it. Yes, I think there may be a race condition at play. But again, that is my best guess :) RAID 6 is the same as what I run so I think you could rule out the actual RAID config itself, but maybe not the capacity of IO of the disks. –  Vasili Syrakis May 8 at 4:35
    
[> ...that is my best guess] Well it's good enough for me. I'm going to build and run these tests. Wonderful lead, thanks so much. –  InteXX May 8 at 4:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.