Back in 2016, We deployed a solution for one of our customers that was an Windows 2012 R2 RDS solution with 2 session host servers, a web server, a couple of file servers, a SQL server and an exchange server.
We've had numerous performance issues since it went live. most of the issues seem to be down to network/communication, but its proving very hard to pinpoint the exact issue.
When we built the solution, we deployed a single 2012 R2 machine and sysprep'd it into a template; we're now of the belief that there was an underlying issue with that install.
The customers internal system is an ASP website with a SQL database. They are constantly reporting speed issues with even the most simple of queries on the database. Sometimes saving notes on a job will actually timeout and fail, when that should be nothing more than an 'INSERT INTO blah blah'.
Also, there have been issues copying files from one server to another. Initially we were able to reproduce an issue whereby the whole server would freeze for a period (sometimes several minutes) when a file was copied from a share on Server 1 to a share on Server 2. We tweaked and updated things and half resolved that, but users still reported that it would not copy files at a decent speed intermittently.
We've also had issues with Outlook on the session host servers reporting random network errors and crashing. Sometimes when you try and reopen outlook it will report that network problems are preventing a connection to exchange. seconds later it will open fine. This again is completely intermittent.
We've troubleshooted pretty much everything we can think of. To name just a few...
- We reinstalled all of the hypervisors.
- We flattened the SAN and rebuilt it.
- We rebuilt the network (10Gb fiber all round).
- We had the hardware vendor (Dell) examine both the SAN and Switches for errors... none were found.
- We nuked and rebuilt the network stack on every single virtual machine.
- We removed NIC teaming.
- We installed every single available update.
I got to the point where I actually re-deployed some of the servers from scratch, this time as Windows Server 2016 to see If I could overcome the issues. Rather annoyingly, after doing so, the corresponding issues were immediately resolved.
For example, When I rebuilt the fileservers, copying files between them is now practically instant - there have been no reports of delays or issues at all. Also, when I rebuilt both the Web server and SQL server, the issue with SQL waiting to process tasks and sometimes failing to save notes etc has been immediately resolved.
I don't really want to go down the route of rebuilding the exchange servers or the session host servers and I've already proven that an in-place upgrade won't do the trick (tried on one of the Fileservers), so I'm hoping that I can identify the root cause of the issue and resolve it.
I'm well aware that I've not provided much in the way of info to confidently point to a culprit, but has anyone come across anything like this before and found the cause or does anyone have an idea what may be the culprit?
I'm planning to raise a support case with Microsoft if I cannot get to the bottom of it shortly... I'll happily pay for the support, however even doing that will require me to be able to scope the issue accurately to ensure it goes to the correct team, and It's very difficult to scope... hell, I'm struggling to pick tags for this post.
Any help or thoughts or suggestions are appriciated.
Thanks in advance.