A previously nicely functioning Remote Desktop Server Farm ahs stopped working two days ago. The setup is as follows:
- DNS resolves "myfarm.mydomain.local" to the IPs of all the farm member servers
- All farm member servers are configured as farm members of farm "myfarm" on Broker MYBROKER
- All farm members are members of the local session broker group on MYBROKER
- Clients are configured to connect to "myfarm"
(All servers involved are virtual Windows2008R2 boxes). Suddenly people started getting the following error (translated from German) and failed to connect.
Unable to connect to terminal server.
The terminal server farm "myfarm" that you are trying to connect to is redirecting you to server "farmmemberX.mydomain.local". Remote desktop cannot verify that this server belongs to the same server farm. This can occur if there is a server on your network with the same name as the server farm. ?It cannot be verified if both remote computers belong to the same remote desktop server farm. You must use the farm name, not the computer name, if you want to mak a connection to a remote desktop server farm.
Contact a network administrator to obtain support if you use a RDP connection that was prepared by the administrator.
If you want to connect with a specific farm member, use "mstsc /admin"
Sometimes it seemed also that they were simply rejected as if having entered wrong credentials (which they have not, most have saved their credentials)
Question 1. Can you explain what is behind this, specifically regarding the "it could not be verified": How does this verification take place if it works? After all, the redirection would not even be attempteed if it were not initiated by broker ...
What we tried: Sometimes it helped to replace the name the client connects to with something else (i.e., we added a name "foo" to DNS that resolved to the same IPs and had users connect to "foo"), but this was far from consistent.
Later we noticed that always the same few servers appeared as "farmmemberX" in the above error message. We experimentally removed these from the farm (at the members themselves and in DNS) and were thus able to reduce the broken eight server farm to a functional two server farm. As this would not be sufficient tor out user load, I wanted to clone one of these; in order to do so I first shut it down and later restarted it - from which moment on it was as bad as the other six servers. Apparently restarting the RDP servers was the fatal thing to do ... According to the logs, this particular server had not been restarted for about two months. So virtually any change made in the last two months could be relevant. Among these are
- We added our first Win2012 AD server into our otherwise Win2003 AD structure
- I recall there were a few cases of IE10/SSL/TLS related security problems that would require manual intervention (regedit and stuff), but am still trying to remember what these might have been
- Tons of windows updates
- Things like invalidated certificates came to my mnd, but I found no such thing
Question 2. Could any of these things cause this problem?
Currently, we dropped the server farm completely, i.e., we only have "poor man's load balancing" via DNS round robin (and we especially miss the reconnect-to-previous-session feature of course)
Main Question. How can I get my farm into working condition again?
EDIT: I should have mentioned that a few clients were lucky and had not peoblems with the RDP farm: those who were still running Windows XP and its older RDP client ...
EDIT after comment: It seems that the main blamed updates KB3002657, KB3035017 were either not installed, or had been installed days before the problem started on the relevant servers (clients, RDP servers, broker, DCs), but I'll try with unsinstalling them anyway ...
UPDATE Some more info:
- I enhanced the event logging on the broker. According to that log, all is fine (no warning) and the session redirect completes normally. It is just that after soe timeout the target session is removed. I tried (failed) in quick connections and in that case, the broker even logged that it tried to reuse an existing session.
- If the target RDP server is set to "RDP-Sercurity" instead of "negotiate", the redirect works (except that the expectable annoying error messages occur to the client)
- I tried a completely new farm (i.e., a different broker with different hosts) and the problem can be reproduced in this system as well. This may suggests that the problem is client-side.
UPDATE with info as requested per comments
If I set security to "TLS 1.0" (instead of "negotiate") at the RDP hosts, the problem persists. If I set to "RDP" the farm works - but everybody has to enter their password twice. In the error situation, for some reason I now often simply get "No connection could be established with the given credentials" instead of the original error. This is accompanied with a login failure event 4625 with status 0xc000006d, substatus 0. Before you ask: All DCs have their clocks in good sync; no LanMan compatabilty settings have been configured in the registry.
The certificates on the RDP host client settings that worked were issued by the still trustworthy internal CA (trusted by all as per GPO) and valid until at least four months in the future. For testing I cheanged these to "automatic" certificates and back, without success.
The original German error message text reads
Von Remotedesktopverbindung kann keine Verbindung mit dem Remotecomputer hergestellt werden.
Vom Remotecomputer "FARMNAME", mit dem Sie eine Verbindung herstellen möchten, werden Sie zum Remotecomputer "FARMMEMBER.DOMAIN" umgeleitet. Es kann nicht überprüft werden, ob diebeiden Remotecomputer zur gleichen Remotedesktop-Sitzungshostserverfarm gehören. Sie müssen den Farmnamen, nicht den COmputernamen, verwenden, wenn Sie eine Verbindung mit einer Remotedesktop-Sitzungshostserverfarm herstellen möchten.
Wenden Sie sich an den Netzwerkadministrator, um Unterstützung zu erhalten, wenn Sie eine RDP-Verbindung verwenden, die vom Administrator bereitgestellt wurde.
Wenn Sie eine Verbindung mit einem bestimmten Fammitglied herstellen möchten, um es zu verwalten, geben Sie "mstsc.exe /admin" an der Eingabeaufforderung ein.
To find out if some recent cleint-side update is at fault, I started with a fresh Windows 7 box and tested after each bunch of updates. It seems the introduction of the first better-than-XP client already causes problems now - but the first such client versions give a different error message (not that it makes sense):
Die Verbindung kann nicht hergestellt werden, da es sich bei dem erreichten Remotecomputer nicht um den angegebenen Computer handelt. Dies kann durch einen veralteten Eintrag im DNS-Cache verursacht werden. Verwenden Sie statt des Namens die IP-Adresse es Computers.