I'm trying to architect a set of web and database servers so that we can load test them. I'm running into some problems given the constraints I have to work with:

  1. We can't load test against production. Duh, mostly. We've looked at a responsible way of doing this, but the bottom line is that it just puts too much risk for downtime for our customers. So this means we have to have an alternate set of hardware to test against.
  2. To protect against what we're trying to protect against, we really need to have truly identical hardware, down to the driver versions running on each box. Merely approximating the bandwidth and software isn't going to do the trick. This means...
  3. The hardware is going to be expensive, so expensive that we can't afford to just have it lying around when we're not load testing it.
  4. This leads to having a temporary setup that we can provision on-demand. The downside of that is that it's still probably pretty expensive, even if it only runs for a week per month, and it has the drawback of being cumbersome and time-consuming. I'm trying to get load testing as a truly first-class artifact in our processes, and this would be a barrier.

If I could wave my magic sysadmin wand, what I would have would be a truly cloned, identical and load balanced architecture, with everything separated from the web application firewall on down to the database server. Thus, when I needed to load test, I could flip a switch that would move one half of the architecture from being load-balanced production into being a load test environment. Even this is not perfect, because it's only half of the capacity, and we're not testing whatever cluster management is going on, but Criminy...you see how crazy this gets.

I'm headed towards the conclusion "load testing on a regular basis just means doubling your hosting costs, period, or else accepting that you're not going to have an accurate hardware test". Is there something I'm missing here?

  • This might be a case where the 'cloud' might give you what you need. Want to load test? Spin up a new instance identical to your prod instance for a few days. – Zoredache Dec 19 '14 at 0:56
  • I know some compagny use SAN replication to run test instance on it. So the actual data is in the SAN in prod. So yes, if you isolate the storage network and the remote network (to be offline) it's doable. So it's more a private SAN network that replicate, and other host that connect to that. – yagmoth555 Dec 19 '14 at 1:16
  • @Zoredache - yeah, we've looked at moving into the cloud, and there are some HIPAA and other security considerations. Not insuperable, but enough to make the process too expensive for our mainline production stack. And without a duplicate hardware stack, we're not guarding against what I'm trying to control. – Chris B. Behrens Dec 19 '14 at 2:18

I suppose it depends on what you are trying to achieve and how your network is configured. You could test against a single server within your stack via a dedicated VIP to understand the load capacity of a single node, this wouldn't give you a definitive result but would give you some idea of the stack capacity.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.