0

I was just reading up on Yoast website here on how the correct robots.txt file should look like with the latest SEO practices. In their example Yoast uses the following:

User-Agent: *
Disallow: /suggest/?*

What exactly does line Disallow: /suggest/?* do?

1
  • Based of the answer below .. is there a particular reason why Yoast uses /suggest/ in their robots file? Is this meant to be just a placeholder or does the suggest actually have a partiular meaning??
    – Joe Bloggs
    Jun 10, 2018 at 0:14

1 Answer 1

1

What exactly does line Disallow: /suggest/?* do?

disallow and ?* will block search engines from accessing any URL that has a ? in it.

Altogether Disallow: /suggest/?* will block search engines from accessing urls such as yourwebsiteexample.com/suggest/?any.php=34

in other words, it will not block search engines from accessing other urls having ?

yourwebsiteexample.com/anyother/?any.php=34 <--- this will be crowed by search engines

more examples https://geoffkenyon.com/how-to-use-wildcards-robots-txt/

3
  • Based on that article, shouldn't it be /*? ? (Based on intuition I thought it was /*?*.) Otherwise, good explanation! Jun 9, 2018 at 15:17
  • Thanks @Aravinda and @EsaJonkien .. is there a particular reason why Yoast uses /suggest/ in their robots file? Is this meant to be just a placeholder or does the suggest actually have a partiular meaning??
    – Joe Bloggs
    Jun 10, 2018 at 0:12
  • suggest should be example directory. no specific meaning. if you dont have one called suggest and if you dont have above requirement to block search engines from accessing any URL that has a ? , u don't have to use it
    – Aravinda
    Jun 10, 2018 at 2:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.