I have used to be in the same situation, suing lighttpd next to apache)for reducing load on apache.
It's better to serve static content with a light web server, as it requires less resources. Also have to mention, that PHP requires apache to run in pre-forked mode, which disables apache from running efficiently. You can distribute the load over to two, differently set up web servers, each handling the traffic it's best in.
Some implementation notes:
You have three options:
- modify your code and segment traffic on IP layer
- don't modify your code and segment traffic on application (http) layer
- get one of the web servers to pass requests coming to the other web server for actual serving
The first is quicker, the second requires less configuration, the third is like a mule.
I would not consider the third option if I were you, as it comes with a configuration nightmare, also, if you misconfigure something at the first web server, nothing will work and it's harder to spot where the problem is.
In the past, I needed to have a solution urgently, so I went with option 2, and utilized a reverse proxy called pound to segment requests upon static/dynamic content and distribute the load to the two different web servers.
Although it works, it requires actively monitoring the http content, which takes it toll on performance (an extra daemon running).
With option 2, you can get better performance with utilizing an extra IP for static content (static.domain.org) and get the clients refer to this static.domain.org for the content. It will still require a reverse proxy, but the proxy does not have to check the Host: header in any of the requests, so it will be quicker.
here is a configuration snippet of pound for your reference: