I have a remote desktop app that was published from a third-party corporation to one of my clients. The remote desktop app connects to a Remote Desktop Gateway Server (RDGW), which forwards the request to a Remote Desktop-serverfarm, where the least used server responds to the request and opens the app. This works perfectly fine for most scenarios.
Today I encountered a situation (let's say the App was published for a new customer), where the RemoteApp would ask the user for his credentials (domain\username and password), right after authenticating the user and accepting the certificate for the RDS farm, mstsc.exe will stop working, being totally unresponsive to cancel or close buttons, and has to be terminated via taskmanager.
While troubleshooting this issue I discovered a difference in the way the RemoteApp connects with the RemoteDesktop Gateway server. In all my positive test scenarios the traffic was completely encapsulated in HTTPS between my MSTSC and the Remote Desktop Server Gateway like this:
The first conversation (as its called in NetMon) is the HTTPS connection to the RemoteDesktop Gateway server (HTTPS, Port 443), followed by a seperate RDP connection, that tries to connect to the RD server itself via its private IP address (I just blanked it in the picture for privacy and security reasons). The second connection eventually fails, because the RD server is not in the same location / subnet, so the private adress is not reachable for the mstsc.
At first I suspected the clients firewall, since it inspects HTTPS traffic, replacing certificates, so it could break a successfull HTTPS connection to the RDGW server. After disabling all webfilter functions on the firewall, the issue still persists. I also tried routing the traffic through a OpenVPN tunnel so it bypasses the firewall completely, sadly without success either.
What could force mstsc not to encapsulate the rdp connection in the HTTPS traffic as in a successfull connection? Is this some kind of fallback strategy or just a plain bug? And what could be used as advanced troubleshooting for the remote app, since it has no valuable log files and the RDS and RDGW servers are not und my control?
EDIT (Added anonymised config file):
redirectclipboard:i:1 redirectprinters:i:1 redirectcomports:i:0 redirectsmartcards:i:1 devicestoredirect:s:* drivestoredirect:s:* redirectdrives:i:1 session bpp:i:32 prompt for credentials on client:i:1 span monitors:i:1 use multimon:i:1 remoteapplicationmode:i:1 server port:i:3389 allow font smoothing:i:1 promptcredentialonce:i:0 videoplaybackmode:i:1 audiocapturemode:i:1 gatewayusagemethod:i:2 gatewayprofileusagemethod:i:1 gatewaycredentialssource:i:0 full address:s:INTERNAL-FQDN-SERVER-HOSTNAME.CUSTOMER.NET <!! EDITED FOR ANONYMIZE REASON alternate shell:s:||name_of_remote_app_on_wts <!! EDITED FOR ANONYMIZE REASON remoteapplicationprogram:s:||name_of_remote_app_on_wts <!! EDITED FOR ANONYMIZE REASON remoteapplicationname:s:name_of_remote_app_on_wts <!! EDITED FOR ANONYMIZE REASON remoteapplicationcmdline:s: workspace id:s:INTERNAL-FQDN-SERVER-HOSTNAME.CUSTOMER.NET <!! EDITED FOR ANONYMIZE REASON use redirection server name:i:1 loadbalanceinfo:s:tsv://MS Terminal Services Plugin.1.RemoteApp-EXT alternate full address:s:INTERNAL-FQDN-SERVER-HOSTNAME.CUSTOMER.NET <!! EDITED FOR ANONYMIZE REASON authentication level:i:2 prompt for credentials:i:0 negotiate security layer:i:1 gatewayhostname:s:external.gateway.ourcustomer.com <!! EDITED FOR ANONYMIZE REASON signscope:s:Full Address,Alternate Full Address,Use Redirection Server Name,Server Port,GatewayUsageMethod,GatewayProfileUsageMethod,GatewayCredentialsSource,PromptCredentialOnce,Alternate Shell,RemoteApplicationProgram,RemoteApplicationMode,RemoteApplicationName,RemoteApplicationCmdLine,RedirectDrives,RedirectPrinters,RedirectCOMPorts,RedirectSmartCards,RedirectClipboard,DevicesToRedirect,DrivesToRedirect,LoadBalanceInfo screen mode id:i:2 winposstr:s:0,3,0,0,800,600 compression:i:1 keyboardhook:i:2 connection type:i:7 networkautodetect:i:1 bandwidthautodetect:i:1 displayconnectionbar:i:1 enableworkspacereconnect:i:0 disable wallpaper:i:0 allow desktop composition:i:0 disable full window drag:i:1 disable menu anims:i:1 disable themes:i:0 disable cursor setting:i:0 bitmapcachepersistenable:i:1 audiomode:i:0 redirectposdevices:i:0 autoreconnection enabled:i:1 remoteapplicationicon:s: shell working directory:s: gatewaybrokeringtype:i:0 rdgiskdcproxy:i:0 kdcproxyname:s: