IP GEo-block and searchmachines

Hello again vedsegaard,

I think we don’t need to worry about SEO because I never have experience that google crawls on back-end and index it. For example, when you install WP into your root directory, https://example.com/robots.txt returns:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

and wp-admin/admin-ajax.php always responses including X-Robots-Tag HTTP header as noindex.

Reference:
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

And wp-comments-post.php returns “405 Method Not Allowed” with blank page, also xmlrpc.php returns the same response with text XML-RPC server accepts POST requests only.

The only I observed is that Bing crawls on my login page whose link is on the front-page. So the only we should take care is wp-login.php and wp-signup.php. You know it’s easy to put a rule into the robots.txt to prevent getting a penalty. But you may not want to expose where wp-login.php is located in the wild.

If you found some crawlers that left their footprints on your back-end, please confirm their IP addresses with whois information. The IP address of a major search engine is able to get its host name by reverse DNS lookup. So you can confirm if someone pretend to be a search engine.

I appreciate you if you onece confirm it.

Reference:
https://www.ipgeoblock.com/changelog/release-2.2.8.html#new-feature-whois
https://www.ipgeoblock.com/codex/ua-string-and-qualification.html#references

And as concerns your solution, you can configure a human friendly error page.

Reference:
https://www.ipgeoblock.com/codex/customizing-the-response.html#human-friendly-error-page

But in this case, a response code is very important. For example, if this plugin returns “200 OK” as a HTTP status code and the spammer receives it, then you’ll get more spams.

An alternative for this is “302 Found“, but unfortunately you can’t configure the URL for redirection because of the same reason as “200 OK“.

So IMO, the best way is to settle the robots.txt if you’d take care of SEO. Or I should provide a function that output X-Robots-Tag HTTP header, should I?

Any discussions are welcomed.

Thanks.

• This reply was modified 8 years ago by tokkonopapa.

And here’re good articles:
https://yoast.com/ultimate-guide-robots-txt/
https://yoast.com/wordpress-robots-txt-example/

Implemented a trial feature which send X-Robots-Tag as noindex, nofollow on blocking at login page in 2.2.9.1.