The tag has no wiki summary.

learn more… | top users | synonyms

8
votes
4answers
2k views

How does Wikipedia generate its Sitemap?

The topic interests me because of Wikipedia's size. It may be easy to create some crons to update the sitemaps periodically in a small site, but what about a big one? So: How does Wikipedia ...
6
votes
4answers
666 views

How can I protected my sitemap index file and sitemap.xml files from leechers?

I have a "content" website that some leechers and 419 scammers love to crawl agressively which also generates costs and performance issue. :( I have no choice: I need to prevent them to access the ...
2
votes
2answers
1k views

How many types of sitemaps are there?

I was perplexed to find two different sitemaps in Google sites: http://sites.google.com/site/(name of the site)/system/feeds/sitemap http://sites.google.com/site/(name of the ...
1
vote
3answers
322 views

Can I protect my sitemap.xml so that only searchengines can download it?

I'm planning of adding a bunch of aggregated lists of pages in my sitemaps that I don't want make it too easy for outsiders to screnscrape. Can I protect my sitemap.xml so that only searchengines can ...
1
vote
3answers
232 views

Unable to generate a sitemap by Google's generator

I would like to generate a sitemap by my Uni.s account such that I have a cron which continuously run the sitemap_gen.py -file. The sitemap is for my site at Google Sites and particularly for the ...
1
vote
1answer
34 views

<head/> in sitemap after moving from apache to nginx

I have a sitemap named http://www.domain.com/sitemap1.php. It starts with this code: <?php echo '<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" ...
1
vote
1answer
32 views

Store sitemaps off-site

We got Nginx webserver. And sitemaps that we generate every week or so ... We migrated to multiple web-servers under single load-balancer lately, and keeping a sitemaps on every webserver seem kinda ...
1
vote
0answers
214 views

How to use sitemap.xml to create a static mirror of a CMS

Is there a tool to create a static mirror of a content management system (CMS) that provides a sitemap.xml file? Ideally, I would point a tool like wget or curl to a sitemap.xml file and have it ...
0
votes
3answers
129 views

Why google didn't crawl all stuff in sitemap.xml?

There are 3000 entries inside sitemap.xml,but it turns out that Google just crawls 300 of them,what's the problem?
0
votes
1answer
145 views

Multiple sitemaps for a website

I have a site that covers several different interest areas - logically, you could think of it as a collection of many 'mini' websites under the MAIN website. Since the areas are not necessarily ...
0
votes
2answers
241 views

Google mini ignoring sitemap

I'm in the processing of setting up a Google Mini device to index our site which has a lot of dynamically generated content. I've created a dynamic site.map file which lists all of the dynamic URL's. ...
0
votes
1answer
184 views

Google Webmaster Tools doesn't like my sitemap

I am having problems with Google webmaster tools and sitemaps with my domain. When setting the sitemap all I get from webmaster tool still is -->Sitemap is HTML ... and I think I found out the ...
0
votes
0answers
63 views

Apache sitemap generator to generate more than 150K URLs & daily 2K URLs

I have a big WordPress installation more than 150K post about 2K daily posts 3 months ago i used Google Sitemap Generator but unfortunatley it is not working with CENTOS 6 It is really very good ...
0
votes
1answer
266 views

Multilanguage google sitemap

Masters, We translated our site to english and im little bit confused about sitemap.xml. Till now, we have a sitemap like this: <?xml version="1.0" encoding="UTF-8"?> <urlset ...
0
votes
1answer
190 views

Cross-submission robots.txt for multiple domains on single host

We are running a site with multiple languages hosted in a single environment on IIS7. For example, oursite.com - english oursite.de - german oursite.es - spanish This is a single-host ...