Spam Bots and Google Bot

  • Hi!

    The description says “By default, the Virtual Robots.txt plugin has a bunch of spam-bots disallowed, the Google bots specifically allowed, ” Can you tell me where this is set?

    Also, I am curious about some of the default settings, Im not sure why one would want robots to crawl directories like “Allow: /wp-includes/js/” or Allow: /wp-admin/admin-ajax.php

    Thank you!

Viewing 1 replies (of 1 total)
  • Plugin Author Marios Alexandrou

    (@marios-alexandrou)

    The description is out of date. Nothing is blocked by default anymore.

    As for the allow rules, the folks at Google have indicated that their crawler should be allowed to access any files references from an HTML page including CSS and JS files.

Viewing 1 replies (of 1 total)
  • The topic ‘Spam Bots and Google Bot’ is closed to new replies.