“Block fake Googlebots” feature blocks indexing of a sitemap

Hi @mpthewho,

Ok,

Thanks for raising the issue, we will investigate in more detail to find the bug and solve it.

WP security > Settings > Advanced settings – IP detection method is correct please make sure and it shows the IP of the https://whatismyipaddress.com/

Google is going to crawl the sitemap URLs. You can cross-check in the Google search console under the Test Live URL (for a property = domain) what error it raises. If you can provide that it will be helpful.

https://snipboard.io/Ea48MF.jpg

Regards

• This reply was modified 7 months ago by hjogiupdraftplus.

Hi @hjogiupdraftplus ,

The test on Googgle search console says this: https://prnt.sc/d_dYdb1K6t-r
The sitemap in Search Console has errors: https://prnt.sc/DBHIe1a42JGX

If I turn of the “Block fake Googlebots” feature, the sitemap is indexed.

Note: All my websites are on Cloudflare.

Regards, Marko

Hi Marco,

https://prnt.sc/d_dYdb1K6t-r

In the Google search console Live URL you should not add a sitemap.xml.

It should be the actual Page URL to be carwaled and displayed as result.

https://prnt.sc/DBHIe1a42JGX

The sitemap says that it does not contain any URL. This should not be the case.

Can you please cross-check in browser {site_url}/sitemap_index.html loads

https://snipboard.io/rZu0CF.jpg

Then you should check the {site_url}page-sitemap.xml loads and of its one of the URL should be checked in Google Search console.

If It shows 403 error one of the AIOS firewall rules might be an issue.

For Cloudflare the IP Detection method must be HTTP_CF_CONNECTING_IP otherwise it might detect proxy server IP and Block fake Google bot login which try check IP address wiht google bots fails.

Regards

Hi @hjogiupdraftplus ,

The sitemap is not empty, it looks like this: https://prnt.sc/tu9sVnoDRcQu

We added the sitemap again, we will see it gets indexed in a few days.

Regards, Mark

Hi @mpthewho,

Ok, keep me posted.

Regards

Got exactly the same issue when decided to enable the same AIOS feature “Block fake Googlebots” for the following domain: https://www.fiderenos.com/en/

Only difference is that I am using RANK MATH SEO plugin an not YOAST but exact same behavior.

For now I decided to stop using this feature since this is the only way for my sitemap to be properly crawled by Google Search Console.

Hi @petrich

Ok, Can you please confirm the google search console does have the https://www.fiderenos.com/ not

https://www.fiderenos.com/en/ inlcuded.

If I cross check robots.txt for https://www.fiderenos.com/en/robots.txt It loads and

Sitemap: https://www.fiderenos.com/en/sitemap_index.xml which shows page not found hope that is not the issue.

https://snipboard.io/vydsTS.jpg

https://www.fiderenos.com/sitemap_index.xml seems correct URL and working.

Regards

Hi support,

We experimented a similar problem.

If “Fake google bots” it set on Google Search Console display an error on /sitemap_index.xml. If I set it off, the error persist. Only if If I reset all the settings of AIOS, and don’t set on “Fake google bots”, the sitemap can ben readed properly by Search Console.

Hi @emerxente,

Can you please cross check in browser the url for sitemap_index.xml load?

If you access it from google search console the server access log what error is?

Is it 403 forbidden or redirect to 127.0.0.1

Hope you have proper IP detection method set at WP security > Settings > Advanced settings – IP detection.

Regards