Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I noticed that my website isn't archived anymore by web.archive.org. When I look at http://web.archive.org/web/*/http://www.cnn.com it is clearly visible that it stopped working in july 2008. web.archive.org has a 6-month-delay policy. This means that 6 month after the crawl run it should be visible to the enduser. So it should have a crawl run from end of 2009 available and not only from mid 2008.

What can I do to get it working again for me and is there a official statement somewhere?

share|improve this question
add comment

1 Answer

The last time their FAQ was updated, the visibility delay was "6-14 months" so it's possible that delay has extended, and you are seeing normal behaviour.

Check that your web site doesn't have a robots.txt file that is telling crawlers not to index it.

That FAQ also lists two other ways to get your site indexed: install the Alexa toolbar in your browser and visit your site; or go to http://www.dmoz.org/ and use the "Suggest URL" form.

share|improve this answer
    
Yes, it is already submitted to dmoz - dmoz.org/News And the robots.txt only tells the crawler to ignore some specific folders. But it seems web.archive.org/web*/foxnews.com also has that problem. So nothing I can do on my server –  Anonymous Mar 5 '10 at 12:13
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.