Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a very strange issue: Google is giving me that some links in my sitemap are not accessible and are giving 404 errors. However, I have tried to open the pages in question and it works fine.

The website has been built using codeigniter and here is a sample of links

Now if you try to open the link above, it will work fine. but if you try to curl -I:

curl -I
HTTP/1.1 404 Not Found

I have made some routes and rewrite rules to exclude the controller name from the URL. In other words, the correct link is

curl -I
HTTP/1.1 301 Moved Permanently
Date: Wed, 29 Feb 2012 18:05:01 GMT
Server: Apache/2.2.14 (Ubuntu)
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1

Apache error log gives

File does not exist: /var/www/myweb/en, referer:

my .htaccess is:

ErrorDocument 404 /index.php
<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteRule ^ar/widwebway/view/([a-z_]{1,50})[/]*$           /ar/$1 [L,R=301,NC]
  RewriteRule ^en/widwebway/view/([a-z_]{1,50})[/]*$           /en/$1 [L,R=301,NC]
  RewriteRule ^en/widwebway[/]*$           /en/home [L,R=301,NC]
  RewriteRule ^ar/widwebway[/]*$           /ar/home [L,R=301,NC]

Any idea of how comes that the page is there but it gives 404 on curl or Google bot?

share|improve this question
up vote 1 down vote accepted

Here's what I found: The browser is actually getting a 404 as well on

When I go to: I get:

Request URL:
Request Method:GET
Status Code:301 Moved Permanently

The move is directed to: This file /en/solutions doesn't actually exist, but with your rewrite rule you're telling the browser to display the contents of /en/widwebway/view/solutions when it sees /en/solutions

From Chrome Network Inspection:

Request URL:
Request Method:GET
Status Code:404 Not Found

Try using the PT flag instead. since the /en/solutions isn't a real file. This should get you away from the 404 errors Example from Apache docs

Alias /icons /usr/local/apache/icons
RewriteRule /pics/(.+)\.jpg /icons/$1.gif [PT]
share|improve this answer
by adding [PT] instead of [L,R=301,NC], i start getting 404 for everything... now i removed all of the rewrite rules, and everything above starts to return "HTTP/1.1 200 OK" ... my concern was to avoid having duplicate content for google.... do you have any idea how to manage it – Alaa Alomari Mar 1 '12 at 7:57
<link rel="canonical" href="">; This should go in the header section of your website. put the link to the page you want people to be visiting in the href. As long as you're using the canonical you can have duplicate content everywhere and Google won't care. – Dez Mar 1 '12 at 15:26

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.