How do I Block Google Bot Crawling Non Existent Images

  • Resolved irenekeal

    (@irenekeal)


    11 months ago

    I have multiple pages that google is trying to call with non existent images how do I block them and any future ones using WordFence please?

    I also have 404 errors which are the requests from Visualsoft-Monitoring (previous website) that tries to search for some non-existing file in the .well-known directory:

Viewing 1 replies (of 1 total)
  • Plugin Support wfmargaret

    (@wfmargaret)

    Hi @irenekeal,

    There is no way to block a bot from crawling specific pages through Wordfence and I would advise against blocking or throttling the Google Bot as it would lead to issues indexing your site on Google.

    You can configure where a bot will crawl to some extent using tools such as robots meta tags or robots.txt.  Google has more information on configuring robots.txt at https://developers.google.com/search/docs/crawling-indexing/robots/intro 

    If you don’t make any modifications, Google crawlers will eventually realize that these URLs are 404s, that they’re not useful to crawl, and they’ll drop out of Search Console over time.  If you want to temporarily throttle crawlers that are hitting too many 404s, you can configure a throttle limit in Wordfence > Firewall > Rate Limiting > If a crawler’s pages not found (404s) exceed.

    Thanks,
    Margaret

Viewing 1 replies (of 1 total)
  • The topic ‘How do I Block Google Bot Crawling Non Existent Images’ is closed to new replies.