I'm more of a LAMP web developer trying to help the sysop. When I joined a project, I inherited some old PHP spaghetti code. Some of that code is that it goes out to a third-party website (let's call it thirdparty.com) and pulls down content with an HTTP-GET request. Unfortunately, the way the code is designed, it needs to do this several times a minute. When we looked at the bottlenecks on the server with 'netstat -a', we saw that connections to thirdparty.com were constantly running when this content would be plenty fine to be gathered once a day.

What I need to know is if the Squid Proxy Caching Server is the solution we need? I'm guessing that this might let us have it pretend to be thirdparty.com on the network. If the web server needs to query thirdparty.com, it hits Squid instead. Squid can then determine whether it needs to supply content from cache or if it needs to go to thirdparty.com for fresh content.

Is this the solution we need? And second, is this easily configured and only to cache thirdparty.com requests?


Squid will work for this purpose, yes.

The only tricky part may be in convincing Squid to cache thirdparty.com if its responses aren't cacheable according to their headers.

If this is the case, have a look at the refresh_pattern configuration directive in Squid; especially the options.


| improve this answer | |

Most of my comments below focus around the assumption that the content is polled on the back-end based on your description.

That code needs to be changed. That's a horrible architecture and has the ability to completely cripple the site with any moderate load. In most cases, it is also rude to the third party Web site.

If DNS for the site goes down, or their site goes down, it has the potential to hold up all of your Apache children while they're sitting waiting for the response. At that point, your site will be unavailable. There's likely even worse implications depending on how the code was written.

I'd say it makes the most sense to poll for the data at a regular interval via a script and use those results. The best place to store that data would probably be a database.

A Web proxy is such a kludgey solution I can't even begin to think it acceptable.

If not already available, a local DNS cache might help slightly reduce overhead.

One possible kludge I see is modifying /etc/hosts for www.thirdparty.com to point to an internal Web server, which displays the content that is pulled down via the aforementioned cron script that runs nightly.

While that might work, you shouldn't do it. The application needs to be changed.

| improve this answer | |
  • Hi Warner. Me again. I think what I'd like to do is at least to just give it a try in an off-hours timeframe. I could drop an entry in /etc/hosts for thirdparty.com to point to the squid server, and then somehow configure squid. For now, I'm testing on my Ubuntu workstation with a local Squid server. I just need to find a good tutorial for just this kind of configuration. Got a link? – ServerChecker Mar 11 '10 at 4:54
  • SQUID would have to be configured as a transparent proxy. – Warner Mar 11 '10 at 4:56
  • Warner: Can that be brought up in a test situation on the same server, and only for one domain, without me having to change the original PHP curl code? – ServerChecker Mar 11 '10 at 5:22
  • Should be technically possible. Different ports/IPs. Be careful. – Warner Mar 11 '10 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.