Prevent crawling of internal site search URLs

  • Resolved trmkr

    (@trmkr)


    1 year, 12 months ago

    Hello,

    I enabled “Prevent crawling of internal site search URLs” setting but there is no change in robots.txt. Can you help please?

    Thank you!

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Support devnihil

    (@devnihil)

    @trmkr If you aren’t seeing your robots.txt as having been updated, it may be due to the file being cached.

    If you clear your cache (server, plugin/theme, browser, or CDN such as Cloudflare) does this resolve the issue? If you are not sure how to clear your server cache you should speak with your host provider.

    We have information on how to clear your browser cache at the following link: https://yoast.com/help/how-to-clear-my-browsers-cache/.

    Thread Starter trmkr

    (@trmkr)

    Thank you! I discovered that if robots.txt already exists, it is not updated. However, if Yoast creates the file for the first time, these lines are added.

    # START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow: /?s=
Disallow: /search/
Disallow: /wp-json/
Disallow: /?rest_route=

Sitemap: https://mywebsite.com/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK

    Regards

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Prevent crawling of internal site search URLs’ is closed to new replies.