How to disable dynamic robots.txt

Viewing 6 replies - 1 through 6 (of 6 total)

  • Are you using any plugin that creates robots.txt? What exactly do you want to block/ allow with it?

    Thread Starter BeeJayEff

    (@beejayeff)

    Not as far as I’m aware – I thought a dynamic one was automatically generated by WordPress. The only plug-ins which I use and which I think could do this are Google Analytics for WP, Google XML Sitemaps and WordPress SEO. But I did disable those and a dynamic robots.txt was still appearing.
    I’ve been advised by my host that the robots.txt might be the cause of Fetch As Google always failing and Googlebot being unable to access my site. I don’t really believe that, but it would be good to cross it off the list of possibilities.

    Moderator Jan Dembowski

    (@jdembowski)

    Forum Moderator and Brute Squad

    If you manually create a robots.txt then that will prevent WordPress from replying back with it’s own version (if you are using fancy permalinks).

    The file can be empty but has to exist.

    Thread Starter BeeJayEff

    (@beejayeff)

    OK, I have created my own, but Google Webmaster Tools still sees the dynamic one. Is that effectively a cached version, and it’s just a question of waiting for it to find the new one ?

    Moderator Jan Dembowski

    (@jdembowski)

    Forum Moderator and Brute Squad

    Link please? And in the file you created can you put a comment such as 

    # My robots.txt file and not WordPress generated

    Just for verification. ??

    If you visit your site and see your robots.txt file the the one Google is seeing is cached on their end. Eventually the new one should be recognized by search engines.

    Thread Starter BeeJayEff

    (@beejayeff)

    OK, done that. Site is at blacklandsalpacas.co.uk.

    Is there no way of telling Google that there’s a new one ?

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘How to disable dynamic robots.txt’ is closed to new replies.