Does Robots Meta replace robots.txt?

  • Does Robots Meta replace robots.txt?

    I realise this may sound like a dumb question, but I can’t find anything in any documentation to answer the question. There are references to Robots Meta conflicting with robots.txt but that’s not quite the same.

Viewing 2 replies - 1 through 2 (of 2 total)

  • The way I understand it…

    The robots protocols are sort-of ad hoc and they have no teeth. They work entirely on a good faith basis. Whether a particular bot reads one, both, or neither depends upon the bot. Likewise, with whether a bot prioritizes one over the other. Its not like, for example, the .htaccess protocols that are enforced by the Apache web server. To defeat .htaccess, you have to defeat Apache or at least a small part of it. Robots instructions just sit in a file and the bot reads them and obeys them, or not, depending upon the bot. And something to think about, you can’t secure information with robots directives, but you can reveal away information with them.

    I don’t think there is a good answer to your question.

    They do different things in my book…

    robots.txt tells a bot where NOT to go.
    sitemap.xml tells a bot where TO go.

    Robots meta tags are just a more vague instruction that cannot go into details about particular folders, but must be put on each individual page.

    Meta tags cannot tell bots not to index images, for example. Robots.txt can.

    I don’t sweat it… I use a sitemap and generate my robots.txt file with a robots generator.
    ??

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Does Robots Meta replace robots.txt?’ is closed to new replies.