Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've noticed that my AWS server occasionally starts using a bunch of CPU for no particular reason, looking something like this:

enter image description here

Observe that it does not occur at specific times, but has a very definite pattern to it. It lasts just under an hour.

Remoting to the machine during this occurrence would invariably make it stop happening. Leaving the account permanently logged on allowed me to capture a more fine-grained CPU usage trace. It looked like this:

enter image description here

That's right; the processes that actually consume that CPU are not in the list. Instead, they appear and disappear all the time. ProcMon was obviously the tool for the job, so I captured a trace. This is what I found:

enter image description here

There's also Postgres involved:

enter image description here

However all the CPU usage is by the Winlogon/LogonUI/etc:

enter image description here

Here's a short excerpt of process start and stop events during this occurrence:

enter image description here

Note that postgres is not interleaved with each start/stop of smss/winlogon/etc, but only some of them.

Any ideas why this happens, and how to prevent it?

share|improve this question
1  
you should check your event logs for time-correlated events –  the-wabbit Feb 25 '12 at 20:47
add comment

2 Answers

up vote 4 down vote accepted
+50

For the postgres part this is because postgres creates a process - not a thread - for each session. This is quite costly on windows ( but rather efficient on unix systems ).

Winlogon / LogonUi part this is rather strange. Is the server remotely accessible? Could there be a network scanner on the network which would try to open port 3389 on the server and thus span a rdp session, which would explain the smss / winlogon / logonui sequence? I think of a network scanner because the session is closed immediately.

So my guess for the bounty: you have a nmap process or some "network discovery" tool which scan ports on your network, or your server is open to the internet without firewall on port 3389 ( and maybe 5432 ).

share|improve this answer
    
What's a "session" in this context - an open connection to the database? –  romkyns Feb 25 '12 at 17:20
    
In the case of postgres a session is a connection to the database. Each remote client creates a process. –  Olivier S Feb 25 '12 at 19:00
add comment

I decided to write a post-mortem; thanks for everyone for their tips; they were spot on!

Tip #1, from syneticon-dj: check the event logs. These spikes were correlated with lots of logon failures, trying usernames like "john", "admin", "test", etc, each one with about 3-5 different passwords. They arrived 3-4 seconds apart.

Tip #2, from Olivier S: this server, being an Amazon EC2 instance, requires RDP. The real problem was that by default, EC2 machines have Network Level Authentication disabled, for some reason. This means that every time someone wants to attempt a password, an entire logon UI is spun up, just to present them with a pretty remote desktop session. This is what caused all the CPU usage.

Tip #3, from me: enable Network Level Authentication (for which there is now another good reason), and change the RDP port away from the default. Just like that, these CPU usage spikes have disappeared completely, and because of the port change my security log is also clean and shows zero failed logon attempts.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.