robots.txt is blocking google from rendering my mobile theme properly

  • Hi, i am totally new to robots.txt, no wonder, it’s a normally non visible file.

    I have 2 sites using both the same totally mobile friendly theme.
    The theme that google says is 100% mobile friendly has only this
    User-agent: *
    Disallow: /wp-admin/
    in the robots.txt,
    whereas the other site is NOT MOBILE FRIENDLY and has plenty else …
    User-agent: *
    Disallow: /wp-admin/
    Disallow: */xmlrpc.php
    Disallow: */wp-*.php
    Disallow: */trackback/
    Disallow: *?wptheme=
    Disallow: *?comments=
    Disallow: *?replytocom
    Disallow: */comment-page-
    Disallow: *?s=
    Disallow: */wp-content/
    Allow: */wp-content/uploads/

    Could anyone let me know which of the disallowed elements would have to be allowed or changed or removed or replaced to give Google access to all css and js files on this site without risking security of the site??

    I found something that points to adding special access to Google only (if i understand this right)
    User-agent: Mediapartners-Google*
    Allow: /

    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    User-agent: Adsbot-Google
    Allow: /

    User-agent: Googlebot-Mobile
    Allow: /
    but it seems 2 of the above are for google ads (?) which i don’t run.

    Also, since this robots file is not visible with FTP, how to edit and than install it properly so that there will be no issue with the virtual robots file from WordPress??

Viewing 3 replies - 1 through 3 (of 3 total)

  • Hi Rudme,

    Google needs to scan your javascripting and css and this may not be blocked by robots.txt. try to remove: wp-content from your robots.txt and test again. Clean cache or cdn if you use this.

    Regards,
    RJ

    For the robots.txt editing you can use seo yoast plugin option for editing robots.txt. If this file is not already there create it from yoast.

    Thread Starter rudme

    (@rudme)

    To all and anyone using WordPress trying to get their site Google mobile friendly and run into the same trouble i did, with a (not seen before, because it’s hidden by default by WordPress for reasons i can not figure out) botched robots.txt that blocks Google’s bots from properly rendering your site, meaning WITH access to style sheets and js files, here is the really simply resolve:
    The plugin
    XML Sitemap & Google News Feeds
    threw all these extra disallows onto my robots.txt file, so beware of that plugin    , as good and nice it maybe otherwise, it sure has no clue about Google spider bots!!

    Of course, if your Google mobile friendly test returns complaints about not being able to access css or js files it may not be the only plugin that dumps strange access denials into robots.txt (for that matter it would be nice of WP to make this file visible in the future so we could all go straight there to check what’s going on).
    It could be any other plugin.
    The old fashioned way to find out which one it is …
    DEACTIVATE one after the other in an order you suspect it could be and check after each deactivation whether the DISALLOW in robotos.txt (which you can always find easily online by going to yourWPsiteurl.pom/robots.txt ) is gone or not yet, until you got the culprit.
    Don’t worry, the deactivated plugins will not need to be completely set up new again once you re-activate them, so this process is no big pain but will most likely solve any Google access issues.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘robots.txt is blocking google from rendering my mobile theme properly’ is closed to new replies.