up vote 1 down vote favorite
share [g+] share [fb]

We're a satellite office of a multinational corporation. We have a local domain controller, but we still experience a crippling performance drop after joining machines to our domain. Does anyone have any suggestions on how we might lessen or mitigate the performance impact of having machines joined to the domain? Please feel free to ask probing questions in the comments should you need additional info.

Note: We do not have full access to the domain or the DC, and thus are unable to effect changes even to our local OU regarding GPOs or general structure/layout etc. However, in the interest of completeness, please post even ideas that may require this level of access.

link|improve this question
50% accept rate
1  
Are you talking about just the joining process or everything once the machine is joined to the domain? – John Gardeniers Jul 23 '09 at 1:58
Hi John. Performance of the machine is what suffers, after the join takes place. I will reword my question to make this clearer, thanks for your interest :) – Ehtyar Jul 23 '09 at 2:21
feedback

5 Answers

up vote 3 down vote

It sounds to me like you've got startup / shutdown scripts that might be running and referencing resources on servers outside your office. I've also seen admins who don't understand DFS and try to do software installations across a WAN.

Run RSoP and see what's applying to one of your machines for startup / shutdown scripts. Odds are good the permissions will be such that you can go out and read the scripts and see waht they're doing. Look for software assignments, too, and see if they're referring to off-site servers.

I'm going to also recommend sniffing the traffic on a PC during startup or whatever other times are slow. Get a box with two NICs, bridge them, and use Wireshark to sniff on the bridge with the PC connected to one NIC and the LAN to the other. You'll get to see what your PC is trying to talk to during those "slow times".

link|improve this answer
+1 for sniffing the network. That will probably reveal the biggest cause of the issue – Matt Simmons Jul 23 '09 at 3:38
Hi Evan, thanks for the answer. You are correct that the US admins have a bad understanding of DFS, which has bitten us on the backside more than once. You are also correct that several startup scripts reference distant servers. Though this does cause a reasonably extensive delay during first login, would it create performance issues around the clock? – Ehtyar Jul 23 '09 at 4:06
@Matt: I don't know why people reach for the sniffer later rather than earlier. I guess, back in the day, when sniffer software wasn't commodity, it had a certain mystique about it. Today it's a tool that I use frequently and with enthusiasm. If I want to see what's going on down there on the wire I just break out Wireshark. The computers want to tell you what's going on-- you just have to listen. I'd much rather look at packets and see what's really happening versus blindfolding myself and only looking at what the software and logs show me on the computers. – Evan Anderson Jul 23 '09 at 4:10
@Ehtyar: In general, there's no specific functionality that should "slow down" a computer after it's joined to a domain. It sounds like you need some good ol' fashioned perf monitoring. Grab Process Monitor, Wireshark, invoke Perfmon on the misbehaving computers and get to monitoring. Unfortunately, that's the only way you're going to get to the bottom of it. – Evan Anderson Jul 23 '09 at 4:14
feedback
up vote 1 down vote

Hrm. That's interesting. What performance are you speaking of? WAN network performance? That might point to AD syncing. LAN performance of the machine being synced? That would make me question your AD size. LAN performance of the local DC? I'm not sure where I'd go with that, honestly.

Or is it just taking forever? If that's the case, it may be inadvertantly querying a remote DC across the WAN.

link|improve this answer
Sorry I wasn't clear enough in my question Matt, thanks for your reply. Running a benchmark on these machines indicates a performance drop in every single category, even things like hard drive response times are negatively effected by joining the domain. I'm aware that there is a certain level of impact to be expected when joining a domain, but the level we're experiencing is simply unacceptable. An Optiplex 755 is barely capable of running Visual Studio, Outlook and SQL Management Studio concurrently... – Ehtyar Jul 23 '09 at 1:53
That sounds suspiciously like some nasty applications being pushed to the PC via startup/login scripts. Are they using monitoring applications? – Dayton Brown Jul 23 '09 at 2:00
Looks like you might be right Dayton. – Ehtyar Jul 23 '09 at 4:11
feedback
up vote 1 down vote

Hmmmm.... It could be that the Relative ID Master is off-network for you. When you create new Computer objects (join the domain, I'm assuming you don't pre-create your objects), it has to go to the RID master. If the DC holding that role is off network for you, it could take a long time to round-trip.

Alternately, it could be that your AD tree doesn't have a Site declared for your satellite network. Therefore, domained machines keep referencing Global Catalog servers not local to you because no site boundary was declared for some reason. You might be able to look that one up yourselves (but not make changes) through the AD Sites MMC tool.

A pair of (very possibly wrong) thoughts.

Edit: Everything? That sounds a lot like the Security checking is horribly slow for some reason. SID lookups and the like are horribly slow. That makes it sound a lot like you're going across the WAN for those lookups when you shouldn't have to. Incorrect sites, or perhaps your local DC doesn't have GC.

How it works is like this:

  • Visual Studio accesses a DLL
  • The OS then determines if Visual Studio can access that DLL at all
    • The User's security token, given to the user a login and contains all their group memberships as well as their direct security identifiers, is then referenced
    • The OS checks the security details of that DLL, finding a list of SIDs
    • The OS then resolves those SIDs against the Domain
      • The domain is probably in Australia for you
    • The OS checks the security principles against the user's security token
  • The OS grants access

Wash, rinse, repeat for every file accessed. The workstation should keep a cache of SIDs, but VisualStudio opens up a metric bajillion of files which may over-run the cache.

You can do a very rough check of domain lookup speeds by right-clicking on any NTFS file or directory and going to Security, entering a real user, and clicking the Lookup User button. How fast that goes should scale with the speeds you're experiencing between your domained/non-domained workstations.

link|improve this answer
Thanks for your reply sysadmin1138 (very cool handle). I can confirm that a site is correctly established for our office. Is there a way for me to find the Relative ID Master box on our network? A DNS record perhaps? – Ehtyar Jul 23 '09 at 2:11
Logically, the DC with the RID Master role would be in the same location as the PDC. So, chances are, it's offsite for you. – phuzion Jul 23 '09 at 2:24
RID pool master isn't something you'll find a record in DNS for, and it's only going to be at play when you're actually creating user objects or computer objects en masse. You can think of RID pool master as handing out booklets of stickers to DC's. Each DC uses a sticker each time a user, group, or computer is created on that DC. When the DC is close to running out of stickers it contacts RID pool master for more-- and that operation involves only a little bit of network traffic. You're not seeing anything related to the RID pool master location. – Evan Anderson Jul 23 '09 at 2:25
1  
@sysadmin1138: RID pool master hands out ranges of RIDs to DCs. Each DC works from that range and, when it's nearing running out, the DC contacts RID pool master and gets more. A round-trip to RID pool master doesn't happen on every security principal cretion. – Evan Anderson Jul 23 '09 at 2:26
Ah, in your answer you said 'off-network', which confused me. In that case, it is definitely offsite. We do have quite heavy latency between ourselves and our primary office (Sydney, AU -> Seattle,US), but surely that can't be effecting machine performance overall? – Ehtyar Jul 23 '09 at 2:31
show 3 more comments
feedback
up vote 0 down vote

Can you verify that the performance is indeed confined to the remote office by possibly taking a fresh machine to the site of the PDC, and joining the domain there? If the performance is still negatively impacted by joining the domain at site of the PDC, it's not your remote office's domain controller.

If the performance is not impacted when joining at the site of the PDC, then you know for sure that it's an issue with your site's domain controller.

Also, for the sake of reference, can you give us an idea of what kind of credentials you have in your domain, so we can base our suggestions on things you will actually be able to do?

link|improve this answer
Sounds like voodoo to me. Where a computer is joined to the domain doesn't impact how it operates in the future. DNS and SRV RR's are used to determine the "best" DC to talk to during bootup and logon. Their "site's domain controller" isn't going to "poison" PCs that have their computer objects created on it. – Evan Anderson Jul 23 '09 at 2:29
My logic is find out whether it's a domain-wide issue of performance, or if it is local to the remote office. If it happens at the site of the PCD, they have a bigger issue to solve, and it's out of Ehtyar's hands. – phuzion Jul 23 '09 at 2:42
Hi phuzion, thanks for the reply. I can't take a machine to the primary office, as it is overseas, though I've heard that users at that office also experience bad performance on their machines. My access to the domain is fully limited to our site and OU, though it is also limited therein with regard to Group Policy, and I have no direct access (RDP etc) to our local DC. – Ehtyar Jul 23 '09 at 4:10
feedback
up vote 0 down vote accepted

After reading the comments from Dayton Brown and sysadmin1138, I had a poke around the startup scripts, and I noticed the installation for an app we use to push software packages to remote machines. Upon further investigation, I found that it apparently does a lot more than push software.

It is a complete IDS and antivirus solution (in addition to Symantec Antivirus for some reason), with about 6 related processes running on each machine. I think it's safe to say that this is causing our performance problems.

My boss has agreed to let me rebuild a machine and prevent it from installing, to test. I'll keep you all posted, thanks for your feedback thus far guys.

link|improve this answer
Is that Symantec Antivirus, or Symantec Endpoint Protection? – sysadmin1138♦ Jul 23 '09 at 3:28
SAV, though we are preparing to make the switch. Currently my machine is the only one in our office running SEP, because I'm "testing" ;) Win 7. Why-any wisdom to impart? – Ehtyar Jul 23 '09 at 3:52
My advice is to avoid SEP like a venereal disease. We used SAV for years, and version 10 worked pretty well for us. Several of our Customers switched to SEP when it came out and had some pretty horrible experiences with it. While it has gotten better with the maintence releases (no more total loss of network connectivity on server computers-- yay!) it's still quite a resource hog. If you can avoid it at all I'd recommend doing so. (Perhaps try to sell the head office on "diversity" in antivirus and get another package, etc.) – Evan Anderson Jul 23 '09 at 4:18
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.