0

I was just reading up on Yoast website here on how the correct robots.txt file should look like with the latest SEO practices. In their example Yoast uses the following:

User-Agent: *
Disallow: /suggest/?*

What exactly does line Disallow: /suggest/?* do?

  • Based of the answer below .. is there a particular reason why Yoast uses /suggest/ in their robots file? Is this meant to be just a placeholder or does the suggest actually have a partiular meaning?? – Joe Bloggs Jun 10 '18 at 0:14
1

What exactly does line Disallow: /suggest/?* do?

disallow and ?* will block search engines from accessing any URL that has a ? in it.

Altogether Disallow: /suggest/?* will block search engines from accessing urls such as yourwebsiteexample.com/suggest/?any.php=34

in other words, it will not block search engines from accessing other urls having ?

yourwebsiteexample.com/anyother/?any.php=34 <--- this will be crowed by search engines

more examples https://geoffkenyon.com/how-to-use-wildcards-robots-txt/

  • Based on that article, shouldn't it be /*? ? (Based on intuition I thought it was /*?*.) Otherwise, good explanation! – Esa Jokinen Jun 9 '18 at 15:17
  • Thanks @Aravinda and @EsaJonkien .. is there a particular reason why Yoast uses /suggest/ in their robots file? Is this meant to be just a placeholder or does the suggest actually have a partiular meaning?? – Joe Bloggs Jun 10 '18 at 0:12
  • suggest should be example directory. no specific meaning. if you dont have one called suggest and if you dont have above requirement to block search engines from accessing any URL that has a ? , u don't have to use it – Aravinda Jun 10 '18 at 2:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.