Does anyone have some stats on the effects of having large amounts (over 100) of redirects in apache conf?

Question

We recently trasferred over a blog to a new site and had to setup a couple of redirects from the old articles to their new location.

Now, the problem here is that we found out that on his old site he had all sorts of different url combinations since he was testing out various seo methods with them.

My question is that I've already grouped a lot of them together and got a good chunk of the site but there are so many 1-off type of urls it is insane. I can't use wildcards and regexes on all of them.

I don't mind putting them as individual redirects in my apache conf but I'm concerned that with 100+ of them it might bog the server down -- does anyone have evidence of this or ran across this problem before?

Answer 1

This may or may not help you. Where I work, we have a ton of domains registered in secondary TLDs (e.g. net, org, various ccTLDs) that generally redirect to a primary TLD (in most cases, com). The net result is around 1,000 URLs that must be redirected elsewhere.

The approach we took was to use Apache's mod_rewrite to generate the redirects, and the configuration is stored in a key:value table (which mod_rewrite can read). The basic configuration would look something like this:

RewriteEngine on
RewriteMap redir dbm=db:/path/to/redir-map.db

# store redir url in a var, or "XX" if not found
RewriteRule ^          -           [E=url:${redir:%{REQUEST_URI}|XX}]

# if the redir url is found, do the redirect
RewriteCond %{ENV:url} !^XX$
RewriteRule ^.*$       %{ENV:url}? [R=permanent,L]

Your redirect map would be a plain text file with lines like this:

# old uri                 new url
/some/path/funny-story    http://newsite.blogtastic.com/funny-story
/some/path/my-birthday    http://newsite.blogtastic.com/birthday-2009

The DB file is built with the following program (included with Apache):

httxt2dbm -v -f DB -i redir-map.txt -o redir-map-db

I like the approach because it keeps our Apache configuration to a minimum, and it does not require us to reload the config when we need to add/change/remove a redirect (we just rebuild the DB file).

In our case, we take about 35,000 hits per day that must be redirected. The server is a CentOS 5.2 VM running under ESX with a single 3 GHz Xeon exposed to it. System load is typically below 0.1, and CPU usage is almost nil. We log the redirects with an additional CustomLog directive, which accounts for the bulk of the server's disk I/O (which itself is still quite low). This server hosts other Web sites as well, so the impact of the redirects is actually somewhat lower than these numbers would indicate.

I can't speak about how this approach compares to using Redirect directives instead of Rewrite, but I can't imagine they perform all that differently. I would not think that adding 100+ redirects would have any noticeable impact on your server.