Okay, lets start off by saying a few things. I came into this with things configured this way and a number of (unknown) steps were taken to get to this point to improve reliability first and performance second.
Now we have problems with both.
The facts:
- Windows 2003 Enterprise Virtual Machine (VMWare, we don't have access to VM Console)
- 8 CPUs (they are newer XEON)
- 16 GB of Ram (I believe 12GB dedicated, rest is burstable)
- 24 GB Page file on its own disk volume (all disks are SAN, connected via fiber)
- Has up to .NET 4.0 although everything is essentially 2.0
- Everything is written with C# in ASMX web services
- This is running an in-house developed application "framework" to support poorly connected users with offline data collection / manipulation
- Data is collected and downloaded to a local SQL Express database based on "last sync date"
- Users can edit offline and then when connected send changes back up
- Each application has it's own app pool in addition to a "framework" app pool
- We have approximately 1700 users distributed across the applications, many have multiple apps.
- approximately 400 "heavy" users using more than one application
- The "framework" app pool has 1 worker process, App 1, App 2, and App 3 all have 10 worker processes.
- Apps 4 through 6 have 1 worker process
- App 1 is a 24x7x365 application, it MUST be up all the time.
- Apps 1-3 are configured to recycle every 240 minutes
- Based on trending data, we have found that if we sustain more than 80-90 "synchronizations" in an hour, we have lots of timeout issues.
- It doesn't appear that many of the worker processes do anything
- I feel we could reduce each to about 4 WP per app pool
- There were lots of issues early on with processes locking up. Not sure on rational for 10 WPs other than that is what they picked and it worked for quite some time
- App 1 during peek times, 50+ syncs an hour, will climb rapidly to around 5.5 GB of memory.
- As things progress, eventually some users get "timeout" errors
- I would rather see 3 processes for that application pool at 2.5 GB each (temporarily) than one holding all of the memory to itself.
- That indicates to me that only one process is handling requests.
- We have tried recycling more frequently, based on memory limits, etc. We get more of the following error:
- System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
- We have had the application pool for App 1 disabled (due to rapid fail protection?) about 3 times this year and App 2 has had this happen once.
What we need to do is get this whole mess under control, better utilize the resources we have, and if possible improve the balancing of this application as quickly as possible.
Ideally I think we need to move to a multi-server situation with a proper load balancer and ASP.NET State server. We do have access to a state server we can use now though, so if there is any benefit to using that across worker processes we can do that. Also want to mention that adding a server (even a virtual one) is not a quick process (6 weeks minimum start to finish for a new VM) and the company is evaluating moving to a new data center so their will be resistance to setting anything new up, especially if it were a physical system.
We do have admin access on the server and can do whatever we need to do as a developer/architect other than rebooting the server. That we have to get special permission for / notify the server monitoring team about the reboot.
I admit this is starting to get beyond my knowledge and we need some guidance. Please ask as many questions as you feel necessary to get a better understanding of what we have going on. Any best practices or general suggestions are also helpful.
I can provide more details about the communication between web services and so on if needed as well.
Edit 1: I should explain a generic "sync". User A connects to the framework server for authentication/logging. Then connects to each App web service as needed. Data is collected from a collection of sources (3rd party WS, Oracle and SQL databases, file shares, etc) and is then passed down to the client using datasets. There are various steps (each a call to the app service to do the work and a call to the framework service for logging) in each synchronization and each is unique to the application.