Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a very strange issue: Google is giving me that some links in my sitemap are not accessible and are giving 404 errors. However, I have tried to open the pages in question and it works fine.

The website has been built using codeigniter and here is a sample of links

Now if you try to open the link above, it will work fine. but if you try to curl -I:

curl -I http://widwebway.com/en/solutions
HTTP/1.1 404 Not Found

I have made some routes and rewrite rules to exclude the controller name from the URL. In other words, the correct link is

curl -I http://widwebway.com/en/widwebway/view/solutions
HTTP/1.1 301 Moved Permanently
Date: Wed, 29 Feb 2012 18:05:01 GMT
Server: Apache/2.2.14 (Ubuntu)
Location: http://widwebway.com/en/solutions
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1

Apache error log gives

File does not exist: /var/www/myweb/en, referer: http://widwebway.com/en/solutions

my .htaccess is:

ErrorDocument 404 /index.php
<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteRule ^ar/widwebway/view/([a-z_]{1,50})[/]*$           /ar/$1 [L,R=301,NC]
  RewriteRule ^en/widwebway/view/([a-z_]{1,50})[/]*$           /en/$1 [L,R=301,NC]
  RewriteRule ^en/widwebway[/]*$           /en/home [L,R=301,NC]
  RewriteRule ^ar/widwebway[/]*$           /ar/home [L,R=301,NC]
</IfModule>

Any idea of how comes that the page is there but it gives 404 on curl or Google bot?

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

Here's what I found: The browser is actually getting a 404 as well on http://widwebway.com/en/solutions

When I go to: http://widwebway.com/en/widwebway/view/solutions I get:

Request URL:http://widwebway.com/en/widwebway/view/solutions
Request Method:GET
Status Code:301 Moved Permanently

The move is directed to: http://widwebway.com/en/solutions This file /en/solutions doesn't actually exist, but with your rewrite rule you're telling the browser to display the contents of /en/widwebway/view/solutions when it sees /en/solutions

From Chrome Network Inspection:

Request URL:http://widwebway.com/en/solutions
Request Method:GET
Status Code:404 Not Found

Try using the PT flag instead. since the /en/solutions isn't a real file. This should get you away from the 404 errors Example from Apache docs

Alias /icons /usr/local/apache/icons
RewriteRule /pics/(.+)\.jpg /icons/$1.gif [PT]
share|improve this answer
    
by adding [PT] instead of [L,R=301,NC], i start getting 404 for everything... now i removed all of the rewrite rules, and everything above starts to return "HTTP/1.1 200 OK" ... my concern was to avoid having duplicate content for google.... do you have any idea how to manage it –  Alaa Alomari Mar 1 '12 at 7:57
    
<link rel="canonical" href="yourdomain.com/content">; This should go in the header section of your website. put the link to the page you want people to be visiting in the href. As long as you're using the canonical you can have duplicate content everywhere and Google won't care. –  Dez Mar 1 '12 at 15:26
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.