up vote 8 down vote favorite
4
share [g+] share [fb]

The topic interests me because of Wikipedia's size. It may be easy to create some crons to update the sitemaps periodically in a small site, but what about a big one? So:

How does Wikipedia generate its Sitemap?

link|improve this question
89% accept rate
feedback

4 Answers

up vote 8 down vote accepted
+50

It's dynamically generatet with a PHP script. For big sites it's probably better to check for changes and only generate if something changed -- or geneare it only all XY minutes/hours/days. Depends on the infrastructure.

The informations needed are all in the database, so it's not such a hard task.

And here is the proof: http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/generateSitemap.php?view=log / http://www.mediawiki.org/wiki/Manual:GenerateSitemap.php

Edit: Ah and this could be also intresting for this Topic:

link|improve this answer
Could you use the PHP-code to generate the sitemap for any big sites? Do you mean by the word "dynamically" that the sitemap is generated somewhat automatically and making slight changes to the code when needed? – hhh Jul 12 '09 at 23:06
Can you clarify the sentence "The informations needed are all in the database, so it's not such a hard task."? Where can I see the database? – hhh Jul 12 '09 at 23:09
I think he means that all of the information is in the database underlying mediawiki. Unless you're one of wikipedia's sysadmins or DBAs, you probably can't get direct access to their DB. – Cian Jul 12 '09 at 23:12
3  
I also think the OP is trying to work out how to generate a Sitemap on a 'large' site, in the case of Wikipedia it is very much RDBMS-driven (MySQL) with all pages being served out of the database. Therefore your DB knows all pages, and you need a simple-ish PHP script (linked above from Subversion) to do it. In the case of other sites, driven by different technologies, then you'll find that the approach needed is different. Last time I checked Wikipedia published their databases for download, or at least, they published their content in a .SQL file. – astinus Jul 12 '09 at 23:23
1  
Here is the [Wikipedia DB Dump][1] :-) [1]: en.wikipedia.org/wiki/… – Gregor Jul 13 '09 at 8:50
show 1 more comment
feedback
up vote 1 down vote

Although the sitemap generation code is in the phase3 trunk, I don't see any evidence that Wikipedia actually has it turned on. The robots.txt file does not point to any site maps.

In either case, Wikipedia runs Squid caches in front of their app servers. They can control how often their sitemap is updated by adjusted the expiry time for the page.

link|improve this answer
+1 smart observation – hhh Jul 12 '09 at 23:52
1  
There's no real reason to expect robots.txt to reference a sitemap, so the absence of such a reference doesn't really prove anything. – John Gardeniers Jul 13 '09 at 0:12
feedback
up vote 1 down vote

I'm not positive, but I think they use the Google Sitemap extension for MediaWiki. This is supported by the Wikipedia page on Sitemaps.

link|improve this answer
feedback
up vote 1 down vote

I was faced with the task to create a site map for our web site a while back. Although it's not the size of Wikipedia, it's still around a hundred thousand pages, and about 5% of them are changed, added or removed daily.

As putting all the page references in a single file would make it too large, I had to divide them into sections. The site map index points to an aspx page with a query string for one of 17 different sections. Depending on the query string the page returns an xml referencing several thousand pages, based on which objects exist in the database.

So, the site map is not created periodically, instead it's created on the fly when someone requests it. As we already have a system for caching database searches, this is of course used to fetch data for the site map also.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.