Robots.txt

  • Resolved yavuzc

    (@yavuzc)


    2 years ago

    I do not have a robots.txt in my root folder, but when I click https://www.destincondorent.com/robots.txt I can see that it is somehow generated. However, I marked a few options in TSF for robots.txt and it does not change anything (such as ignore pagination)

    The page I need help with: [log in to see the link]

Viewing 1 replies (of 1 total)
  • Plugin Author Sybre Waaijer

    (@cybr)

    Hello!

    TSF does not use robots.txt for blocking because it is bad practice. By parsing each request to your website individually, TSF uses canonical URLs and meta tags to achieve this much more robustly. This method ensured compatibility with every theme and plugin (provided that they abide by WordPress API standards).

    To learn more, please see https://developers.google.com/search/docs/advanced/robots/intro.

Viewing 1 replies (of 1 total)
  • The topic ‘Robots.txt’ is closed to new replies.