Robos.txt and BOM error in google search console

  • Resolved cpucpu

    (@cpucpu)


    7 years, 3 months ago

    I was aparent to me that googlebot wasn’t following the rules on the robots.txt file for a while. I checked the search console and there were multiple errors in said file.

    This is happening to me: https://www.carlj.ca/2014/05/05/webmaster-tools-robots-txt-user-agent-syntax-understood/

    I applied this temporarily fix:

    
/* Preppend text to robots.txt */
add_filter('the_seo_framework_robots_txt_pre', function($defaults, $args = array() ) {
    return "#\n";
}, 10, 2 );

    A “syxtax not understood” error persists in the first line. However I expect googlebot to work for now.

Viewing 5 replies - 1 through 5 (of 5 total)
  • Plugin Author Sybre Waaijer

    (@cybr)

    Hi @cpucpu,

    I’m glad you’ve been able to find a resolution beforehand, awesome!

    Where did you see the “syntax not understood” notice?

    I’m wondering if this is related to a caching/minification plugin…

    Cheers!

    Thread Starter cpucpu

    (@cpucpu)

    Hi, I checked it in: https://www.google.com/webmasters/tools/robots-testing-tool I suppose this could affect robots.txt files generated by the SEO framework. It was a major issue for me, well it still is since googlebot still needs to un-index dozens of directories that i disallowed week ago.

    Plugin Author Sybre Waaijer

    (@cybr)

    Hi @cpucpu,

    It was a major issue for me, well it still is since googlebot still needs to un-index dozens of directories that i disallowed week ago.

    That is disconcerting.

    The problem debugging this is that The SEO Framework doesn’t initiate the virtual robots.txt output, WordPress does.
    TSF only alters the contents, and in doing so, it can’t touch the BOM.

    You could try to see if there’s another plugin conflicting (with your fix disabled), or temporarily disable TSF to see if Google accepts it.
    This is an edge-case, so I can’t provide any more insights. If you happen to find the causal plugin, please do share ??

    Thread Starter cpucpu

    (@cpucpu)

    Thank you, I get your point. So you’re not experiencing this issue? I mean does your .txt file pass the google console test. I’ll do more testing to see what else could be causing this.

    • This reply was modified 7 years, 3 months ago by cpucpu.
    Plugin Author Sybre Waaijer

    (@cybr)

    Hi @cpucpu,

    As I work closely with Google Search Console alongside development on multiple sites, I can attest the BOM issue is (AFAIK) unique ??

    Disabling plugins one by one helps pinpointing the plugin guilty of this result. Note that if two plugins try to do the same thing, issues might arise too; so, you might find a combination of plugins that cause this rather than just one on its own.

    i.e. if disabling plugin A helps, but leaving plugin A enabled and disabling plugin B also works, then there’s a conflict between the two.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Robos.txt and BOM error in google search console’ is closed to new replies.