Unwanted URLs are crawling

Viewing 4 replies - 1 through 4 (of 4 total)
  • Suwash

    (@suascat_wp)

    @kanojia

    I would recommend you to manually create a rule in your site’s robots.txt file to block the search feed URLs. — https://yoast.com/ultimate-guide-robots-txt/

    Alternatively, Yoast SEO Premium provides an inbuilt option to enable blocking the search feed from crawling by the search engines.

    Read more: Search results feeds

    Thread Starter kanojia

    (@kanojia)

    Could you tell me rules which need to be added for such urls to remove?

    Thanks

    Plugin Support Maybellyne

    (@maybellyne)

    A mistake in robots.txt can have unintended consequences, so I can’t provide such directives. Since the URLs are reported as crawled but not indexed, and have a noindex tag, there’s no cause for alarm.

    Plugin Support Maybellyne

    (@maybellyne)

    This thread was marked resolved due to a lack of activity, but you’re always welcome to re-open the topic. Please read this post before opening a new request.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Unwanted URLs are crawling’ is closed to new replies.