autoptimize Style sheet blocked by robots.txt file

  • Resolved geekgyan

    (@geekgyan)


    8 years, 4 months ago

    I was using the fetch as Google function and found out that the page visible to google bot and the actually page a vistor sees are diffrent.
    futher down when i looked for blocked resources i got 1 major severity issue that the autoptimize stylesheet from wp-content/cache folder was blocked by robots.txt file.
    Image1
    Image2
    I have read many articles which instructed to Disallow: /wp-content/cache
    from robots.txt file.
    So should i ignore this issue or should i remove the line Disallow: /wp-content/cache from robots.txt file

Viewing 5 replies - 1 through 5 (of 5 total)

  • hmmm .. maybe you could disallow wp-content/cache, but allow wp-content/cache/autoptimize?

    Thread Starter geekgyan

    (@geekgyan)

    ok will try

    Thread Starter geekgyan

    (@geekgyan)

    Heres my robots.txt file> should i add Allow: wp-content/cache/autoptimize ?

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /search/
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Disallow: /readme.html
    Disallow: /xmlrpc.php
    Allow: /wp-content/uploads
    Allow: /wp-admin/admin-ajax.php

    User-agent: NinjaBot
    Allow: /

    User-agent: Mediapartners-Google*
    Allow: /

    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    User-agent: Adsbot-Google
    Allow: /

    User-agent: Googlebot-Mobile
    Allow: /

    ALso There is one more problem now,the autoptimize stylesheet css is taking 2 secc to load. Here’s the image form pingdom test
    Image

    should i add Allow: wp-content/cache/autoptimize

    guess so (I’m not a robots.txt expert), but add a slash before wp-content;
    Allow: /wp-content/cache/autoptimize

    ALso There is one more problem now,the autoptimize stylesheet css is taking 2 sec to load.

    Your autoptimized CSS is served as a static file, so this is just between your webserver & the browser (no PHP, no WordPresss, no Autoptimize) so there’s nothing much AO can do. But I wouldn’t worry; according to this webpagetest.org test (well, 5 tests), it generally loads in 700ms ??

    frank

    Thread Starter geekgyan

    (@geekgyan)

    ok thankyou i have now actually allowed cache in robots.txt. I wonder y people suggest to block it beacause many caching pluging actually say that bots should hit cache so that ranking incrases
    Anyways thank you. And awesome plugin ??

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘autoptimize Style sheet blocked by robots.txt file’ is closed to new replies.