Are Google Bots Blocked?

  • Resolved pj220

    (@pj220)


    3 years, 2 months ago

    Hello, I want to use this plugin to block brute force bot attacks. I am concerned that I might also block Google Search Bots from indexing my websites blog posts. Does this plugin block Google Bots? or are they free to scan my site to index in Google Search?
    Thanks

Viewing 3 replies - 1 through 3 (of 3 total)
  • Thread Starter pj220

    (@pj220)

    Hi Darko,

    hope you are doing well!

    How can I make sure IP Location Block plugin does not block any Google Search Bot Crawlers that I want to Whitelist/Allow to come to visit my website? – for search engine ranking?

    see

    https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot?visit_id=637689850477577393-2160476811&rd=1

    whitelist/allow
    googlebot.com or google.com

    `Verifying Googlebot
    You can verify if a web crawler accessing your server really is Googlebot (or another Google user agent). This is useful if you’re concerned that spammers or other troublemakers are accessing your site while claiming to be Googlebot. Google doesn’t post a public list of IP addresses for website owners to allowlist. This is because these IP address ranges can change, causing problems for any website owners who have hard-coded them, so you must run a DNS lookup as described next.

    Verify that Googlebot is the crawler

    Use command line tools

    Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.

    Verify that the domain name is either googlebot.com or google.com.

    Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name. Verify that it’s the same as the original accessing IP address from your logs.

    Plugin Author Darko G.

    (@darkog)

    Hi @pj220 ,

    Googlebot is allowed by default and there is matching option through their User Agent.

    In any case please check Settings > IP Location Block > Front-end target settings > UA string and qualification section.

    You should have something like this:

    
Google:HOST,bot:HOST,slurp:HOST
spider:HOST,archive:HOST,*:FEED
embed.ly:HOST,Twitterbot:US,Facebot:US

    Best Regards,
    Darko

    Thread Starter pj220

    (@pj220)

    Thank you!

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Are Google Bots Blocked?’ is closed to new replies.