Do i need to add a robots.txt file

  • Resolved stephenturner

    (@stephenturner)


    10 years, 3 months ago

    Hi guys, can someone please set me straight. I started using wordpress a few years ago, but some of my websites would not index in google, it said it couldn’t find a valid robots.txt file. I since use the robot.txt plugin which creates a robots.txt file, i don’t add anything to it, just leave it to the default setting:

    User-agent: *
    Disallow: /

    However, for some sites google still couldn’t find the robots.txt file, sop i deleted the line Disallow: /, now every website i do is found in google.

    so can someone explain:
    1. Why i have to create a robots.txt file (and where can i find this on my server), when i have been told wordpress creates on for me and…
    2. why do i have to delete the second line (Disallow: /) on some sites to get google to index them?

    hope someone can explain, cheers guys… ??

Viewing 4 replies - 1 through 4 (of 4 total)

  • Is it possible in your Dashboard, under Settings –> Reading –> Search Engine Visibility box was checked? This would cause the robot.txt file to have created the line Disallow: /

    If this box is checked, Google in most cases will respect that and not crawl the site. Perhaps this was the issue and I hope this helps.

    Thread Starter stephenturner

    (@stephenturner)

    thanks for your answer, does the second line disallow: / mean not to index anything, ie google won’t crawl the whole site? If so, this would make sence.

    secondly, the plugin i use (robots.txt plugin) creates a section in my settings in the reading to amend the robots.txt file, but is this a physical file (not the virtual wordpress file), if so, where is this file, i cannot find it on my server anywhere?

    thanks again in advance… ??

    To the first part that is correct.

    I’m not familiar with the plugin you’re using, so I’m not entirely sure what exactly is or isn’t happening. Perhaps the file is hidden file on you’re server, in which case you may want to check with your specific host to see if they can help you locate the file?

    Thread Starter stephenturner

    (@stephenturner)

    thank you… one last question, i know wordpress automatically creates a virtual robots.txt file, is this good enough for google, if there is no physical robots.txt file how does google see the virtual file? When the crawlers look into the directory, apparently it reads the virtual robots.txt file, but it doesn’t actually exist so really my question is:

    is the wordpress virtual robots.txt file good enough, so in future i need not bother installing a robots.txt plugin as i don’t add anything into the file anyway?

    thanks… ??

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Do i need to add a robots.txt file’ is closed to new replies.