prevent from Google crowler

  • Resolved chrishd06

    (@chrishd06)


    1 year, 4 months ago

    I woke up this morning and my homepage was not ranked anymore after setting up the rate limit as per Wordfence guidline.
    Wordfence blocked the google bots by returning a 403.

    Can i link the google crawlers Json files to wordfence to be white listed ?
    because adding all IP is very timeconsuming.
    Thanks

Viewing 1 replies (of 1 total)
  • Plugin Support wfpeter

    (@wfpeter)

    Hi @chrishd06, thanks for reaching out to us about this.

    When you say set up as per Wordfence guideline, did you check “How should we treat Google’s crawlers” here: https://www.wordfence.com/help/firewall/rate-limiting/

    Just ensure your rules already don’t limit Verified Google crawlers, and throttle instead of block. Throttling is generally better than blocking with crawlers because any good search engine understands what has happened if it is mistakenly blocked and your site isn’t penalized because of it. You can see my Rate Limiting settings if you click that link.

    I hope that helps you out!
    Peter.

Viewing 1 replies (of 1 total)
  • The topic ‘prevent from Google crowler’ is closed to new replies.