If you're actually going from a single-server environment to a load-balanced cluster of 10, that sets of warning bells for me instantly. I have a bunch of questions which you hopefully have already figured out, but I'll point them out anyways, and then provide some generic considerations.
How did you arrive at the number 10? Why are you not scaling to 2 or 3, and adding more as needed?
Why are you going to load balancing in the first place? Eg, are you going for high load, high availablity, or both? Is there an immediate need, or is this a predictive need?
If you're under load now, and trying to scale to solve it, one big question I'd ask is if you've actually identified the bottleneck. You mention you're using .NET, and SQL for session state, so I'd guess that you're also working with a SQL-backed application. Are you also balancing the SQL server? Can the SQL server handle 10x the connections you have now?
If you're going for availability, have you considered all other points of failure? Do you have redundancy on your load balancer? Do you have redundancy on your database server(s)? Do you have redundancy on your internet uplink (consider all points: single cable, single switch, etc)? For availability, you're only as safe as your weakest link. It doesn't matter if you have 10, or even 100 front-end web servers, if you only have one database server and it goes down.
A lot of the time your bottleneck will be your database server. If that's the case, it doesn't matter how many front-ends you have.
If you're using SSL, there are two typical modes load balancers usually operate in, which affect how SSL works:
- Layer 4: This is TCP level. SSL is handled by each server, and thus the SSL cert has to be installed on each server.
- Layer 7: This is application level, also called reverse proxy. The load balancer handles the HTTP session, and makes a second connection to the application server. In this mode, the SSL cert is installed only on the balancer, and the connection to app servers is typically HTTP. This is sometimes called "SSL Offloading", and is typically useful if your load balancer is powerful, and you don't want your app server handling the encryption overhead of SSL (eg. your app is CPU-intensive).
Ensure your balancer is set up to take servers out of rotation when they go down, and test this functionality. You should be able to down servers without it affecting the rest of the servers. Note PING vs HTTP vs response time checks. (Pinging doesn't mean HTTP is responding)
Load test your environment. You may not be able to go full-out, but you should be able to load down at least a couple servers (with just those two in the balancer).
Run a staging environment. This may not be 10 servers, but should be enough to replicate the production system for deployment testing.
Have an automated deployment script, and be very strict about source management and configuration management. Ideally this means EVERYTHING (including config files) is in a source control system, and you have automated builds to create everything right up to your installer/script.
Get a tool that monitors both your external-facing site, and all internal servers. If a server dies, nothing really changes from the perspective of the outside world, but you want to know and fix the server anyways. If you do start having issues with performance or availability, what you don't want to find is that 3 servers have been down for a month and the rest are struggling with the extra load.