1

We're developing our web application and have been working on reducing our load times. When we began developing the application, we signed up with a well known hosting provider offering "dedicated solutions" on the cloud. As a result, we have one dedicated server for our web application and one dedicated server for our database. Both servers were setup with the same amount of RAM, the same CPU, and SSD drives. Even with investing time & money into our infrastructure, we noticed 8-10 second load times.

We were advised by another hosting company that we should scale down and put everything onto one server (instead of splitting it up). They mentioned that having the servers separate would lead to higher load times because of network latency and stated that PHP would communicate quicker with SQL via sockets than over the network. This I know to be true, but I did not expect the results we got. Immediately after moving everything to our original application server, our load times dipped to 3-4 seconds from 8-10!

The problem is that we're now looking into a new hosting provider and have been advised to utilize a cluster with a load balancer, database server, application server, and scale from there. The concern is that if we separate the application and database server again, that we'll be back to square one.

From everything I've read, it seems that it's almost always recommended to have these servers split rather than grouped together. Is there a performance boost that comes with doing this when it is setup properly, or is it merely for scalability in the long run?

I appreciate the assistance!

2

Your question is really broad, so I just mention a few aspects:

  • local socket I/O is faster than TCP, but for most applications this should be miniscule compared to all other parts of your turnaround time (load balancer, PHP processing, DB query processing, ...)

  • split systems allow for better caching, e.g. the DB server can keep more indices in RAM.

  • possibly a scalability point: split systems are easier to configure, e.g. to deploy a new software version or update PHP you could just add a new application server, test it, and finally remove the old one.

  • to investigate your problems: check how many DB connections are opened for every web request. One explanation for your measurements would be an application that uses many SQL queries without persistent connections, so a new TCP connection is opened for every DB access.

|improve this answer|||||
  • Hey there! I appreciate the assistance! I apologize for the broad question -- basically I was trying to confirm that having a split environment is better than a singular server for performance and stability. When we saw our own performance improve by consolidating the environments, our team began to do research and found a well known consultant whom told us to use one server instead of multiple nodes. That contradicts what everyone else seems to say, so we were confused. Your response sheds some light and gives me direction to investigate further. Thank you! – Brian Schroeter Feb 11 '14 at 22:53
1

I don't know what you're doing to get 8-10 second load times (assuming you define "load time" as "time between the HTTP request arrives and the page is built and send to the browser").

You shouldn't be able to get your CPUs to 100% usage with a web server and a database, and even if you manage this somehow, putting web server and database on one server wouldn't help.

Also, any kind of overload on the DB server wouldn't be alleviated by moving both servers on the same hardware.

So the problem almost has to have something to do with

  • very many very small SQL statements that are sent to the DB individually, so even the small latency on a local network accumulates (imagine you have 10000 SQL statements per page, and a network latency of 0.1 msec. This will result in your 10 second load time).
  • huge blobs stored in the database that need to get to the web server through the sql connection, which is typically slower than a protocol designed for file transfer, especially over the network
  • the network connection between your hosts is artificially limited in some way

Maybe it's something else that i can't imagine at the moment, because it's very uncommon for a typical web application to become slower when you distribute it over more CPUs, as long as those CPUs have a fast network connection between them.

As long as you don't find out what causes/caused the trouble on separate hosts, you might have the same problem again, or you might not.

I had the first kind of issue last year - a customer of mine contracted a 3rd party to develop some software for them. A typical operation on a demo laptop took about 4 hours to complete. When my customer moved the software to the intended production environment (BIG application server, BIG database cluster), the same thing took a little over 16 hours. Sifting through the logs and doing a network trace, we found out that the application had did about 15.000 selects per seconds on the dev system, and the 0.3 ms latency between application server and database limited this to slightly over 3000 selects per seconds. The devs were told to change the way they accessed the database (do a join on 2 tables instead of a select on one, then a single-row-select on each of the results), which resulted in the whole operation take less then 30 minutes.

The point is, the kind of trouble you're having is unusual, might have to do with your software behaving in an unusual way, and you really should investigate what's going on here and why the 2-machine-setup was so much slower.

Splitting into 2 machines should normally improve performance, because you have more CPUs to do the job. It also increases maintainability. Your database might need a special kernel parameter or patch level to work well; your web server might have conflicting requirements. And, whenever you do an upgrade, it's so much easier to be able to upgrade one of the two systems without touching the other.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.