0

Okay, lets start off by saying a few things. I came into this with things configured this way and a number of (unknown) steps were taken to get to this point to improve reliability first and performance second.

Now we have problems with both.

The facts:

  • Windows 2003 Enterprise Virtual Machine (VMWare, we don't have access to VM Console)
  • 8 CPUs (they are newer XEON)
  • 16 GB of Ram (I believe 12GB dedicated, rest is burstable)
  • 24 GB Page file on its own disk volume (all disks are SAN, connected via fiber)
  • Has up to .NET 4.0 although everything is essentially 2.0
  • Everything is written with C# in ASMX web services
  • This is running an in-house developed application "framework" to support poorly connected users with offline data collection / manipulation
    • Data is collected and downloaded to a local SQL Express database based on "last sync date"
    • Users can edit offline and then when connected send changes back up
  • Each application has it's own app pool in addition to a "framework" app pool
  • We have approximately 1700 users distributed across the applications, many have multiple apps.
  • approximately 400 "heavy" users using more than one application
  • The "framework" app pool has 1 worker process, App 1, App 2, and App 3 all have 10 worker processes.
  • Apps 4 through 6 have 1 worker process
  • App 1 is a 24x7x365 application, it MUST be up all the time.
  • Apps 1-3 are configured to recycle every 240 minutes
  • Based on trending data, we have found that if we sustain more than 80-90 "synchronizations" in an hour, we have lots of timeout issues.
  • It doesn't appear that many of the worker processes do anything
    • I feel we could reduce each to about 4 WP per app pool
    • There were lots of issues early on with processes locking up. Not sure on rational for 10 WPs other than that is what they picked and it worked for quite some time
  • App 1 during peek times, 50+ syncs an hour, will climb rapidly to around 5.5 GB of memory.
    • As things progress, eventually some users get "timeout" errors
    • I would rather see 3 processes for that application pool at 2.5 GB each (temporarily) than one holding all of the memory to itself.
    • That indicates to me that only one process is handling requests.
  • We have tried recycling more frequently, based on memory limits, etc. We get more of the following error:
    • System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
  • We have had the application pool for App 1 disabled (due to rapid fail protection?) about 3 times this year and App 2 has had this happen once.

What we need to do is get this whole mess under control, better utilize the resources we have, and if possible improve the balancing of this application as quickly as possible.

Ideally I think we need to move to a multi-server situation with a proper load balancer and ASP.NET State server. We do have access to a state server we can use now though, so if there is any benefit to using that across worker processes we can do that. Also want to mention that adding a server (even a virtual one) is not a quick process (6 weeks minimum start to finish for a new VM) and the company is evaluating moving to a new data center so their will be resistance to setting anything new up, especially if it were a physical system.

We do have admin access on the server and can do whatever we need to do as a developer/architect other than rebooting the server. That we have to get special permission for / notify the server monitoring team about the reboot.

I admit this is starting to get beyond my knowledge and we need some guidance. Please ask as many questions as you feel necessary to get a better understanding of what we have going on. Any best practices or general suggestions are also helpful.

I can provide more details about the communication between web services and so on if needed as well.

Edit 1: I should explain a generic "sync". User A connects to the framework server for authentication/logging. Then connects to each App web service as needed. Data is collected from a collection of sources (3rd party WS, Oracle and SQL databases, file shares, etc) and is then passed down to the client using datasets. There are various steps (each a call to the app service to do the work and a call to the framework service for logging) in each synchronization and each is unique to the application.

1 Answer 1

1

It sounds as though, the scalability issues are at the application server and not the database server. Therefore, the best bet is to load balance the application servers.

Load balancing is pretty easy and can be done at the networking level. This allows you to put X number of physical or virtual machines behind a VIP and distribute the load across the machines unilaterally. A given user will hit different machines on each request.

When load balancing, its best to avoid session state information. However, if you can't avoid it, a session state server will be needed that all the machines can connect to. This is fairly trivial in .NET since session state providers can be easily plugged in using the Web.config. This allows the application servers themselves to be stateless and therefore, the VIP can direct incoming traffic to any server it wants.

Load balancing will also allow you to take a machine out of service if it becomes problematic or maintenance needs to be deployed. This also comes in handy during deployments as you can take half of the machines out of service and make a deployment and then swap them. This prevents having 2 version of application code running at the same time.

2
  • Hey I know you... Mar 24, 2011 at 17:10
  • I think you are right... but I don't know WHEN we can get there. I am sure if we made a strong case for it we could do that. Give your familiarity with what I am dealing with, do you have any other specific advice? Mar 24, 2011 at 17:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .