4

For the past week my users have been experiencing extreme logon times. They are generally in the 20 minute range or more. I will lay out my research below to see if anyone may know how to find the root cause.

Terminal Server:

  • x2 Quad core Xeon 2.6Ghz processors
  • 16 GB ram
  • x2 300GB SAS drives in Raid1
  • Gigabit Lan
  • Windows 2k3 Enterprise
  • Average users connected 23

Domain Controler:

  • Windows 2k3 Enterprise
  • Gigabit lan
  • Also stores roaming profiles

Most of the information I have gathered has been from parsing the USEREVN.log file. I wrote a custom parsing tool to try and glean as much information as possible from logging the users login processes. Here are some stats from those logs.

  • The single most logged routine is ReconcileFile
  • The most time spent total is on the routine ReconcileFile
  • There is no single routine that is taking an extreme amount of time.
  • ReconcileFile and RecurseDirectory take 7 times longer than GPO processing
  • The average time spent per ReconcileFile routine is 223 milliseconds with normal user load
  • the average time spent per ReconcileFile routine is 220 milliseconds when only 1 user is accessing the system
  • From a user prospective the average execution time of a thread handling a routine for that particular user can be upwards of 1 second, though, the average time per routine still may be about 223 milliseconds.
  • A particular user may have routines being executed to log them on over a 20 minute period but actually only have around 2 minutes of execution time devoted to their login.

Having gathered all that diagnostic data I still can't find the root cause of the issue. The servers resources have not changed between this week and last. So, I'm at a loss of were to look next.

  • 1
    I'm impressed by your work gathering data. I find the USERENV.LOG to be frustratingly verbose but low on detail. If I were you, I'd also run a Process Monitor trace on a user logging-on to see how that looks, too. I suspect that's going to help you pinpoint the root cause. – Evan Anderson Oct 24 '14 at 0:43
  • 1
    1) If you haven't changed the servers, what else might have changed? Is there AntiVirus with auto-update on either server? Would they have installed Windows Updates? Has someone plugged something in which is saturating or otherwise breaking the network (e.g. a cable loop)? 2) Divide-and-conquer troubleshooting: If you create a fresh user account, is that slow to logon? Can you make a user exempt from roaming profiles so they only logon locally and compare? – TessellatingHeckler Oct 24 '14 at 0:57
  • 1
    Are you redirecting My Documents or any other folders? – joeqwerty Oct 24 '14 at 0:59
  • 1
    So this morning I did my routine logging and am floored. The avg time of the ReconcileFile routine is down to 43ms. I did not do anything to the server. The only change to the entire network was we swapped out a core switch that doesn't even interconnect the terminal server to the domain controller / profile storage location. Sigh... – Donovan Oct 24 '14 at 17:23
  • 1
    After a few more days of testing I still don't have any more answers as to what happened. After we swapped out that switch (which wasn't swapped due to any failure) my users are back to normal login times. Today I recorded a 23ms average time on the ReconcileFile routine. I'm really not sure what to make of it. I guess the lesson here is throw a dart at the network dartboard if you have excessive login times? – Donovan Oct 27 '14 at 22:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.