Google cannot access my robots.txt!

  • Hi folks!
    I have a WP site installed in a directory called /wp.
    I followed the official tutorial to move site from sub-directory to root directory by changing the wp path and adding a index.php file in root directory which includes all necessary files.
    Now, I have some troubles indexing my WordPress site on google webmaster tools. I added my site with this first version of .htaccess

    <IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

    Now, I tried to add these lines to .htaccess file, but Google still does not index my site.

    # BOT SETTINGS
SetEnvIfNoCase User-Agent .*google.* search_robot
SetEnvIfNoCase User-Agent .*yahoo.* search_robot
SetEnvIfNoCase User-Agent .*bot.* search_robot
SetEnvIfNoCase User-Agent .*ask.* search_robot

Order Deny,Allow
#Deny from All
Allow from env=search_robot

    Where is the error?

Viewing 7 replies - 1 through 7 (of 7 total)
  • Moderator James Huff

    (@macmanx)

    It’s really hard to say without seeing the site. What is the URL of your site?

    Thread Starter tab7x

    (@tab7x)

    Hi,
    the link is radio_DOT_uniroma3_DOT_it
    Thank you for your support!

    Moderator James Huff

    (@macmanx)

    Robots.txt file is publicly accessible, and here is the contents:

    User-agent: *
Disallow: /sito/wp-admin/

    You’re just blocking bots from /sito/wp-admin/

    Head back over to https://www.google.com/webmasters/tools/robots-testing-tool and try testing the site’s robots.txt file there.

    Thread Starter tab7x

    (@tab7x)

    The problem is just there!
    Google cannot retrieve robots.txt but if I test robots.txt availability from robots-testing-tool page the result is “allowed”.
    It might be a problem of apache configuration?
    I am not on a shared hosting but on a debian VM

    Moderator James Huff

    (@macmanx)

    If Google Webmaster Tools says it’s allowed to access the robots.txt file, where are you getting the notice that it can’t retrieve the file?

    Is it an old message from Google Webmaster Tools? If so, those aren’t in real-time. They come in incidentally, like email. You have to delete the message once reviewed, it won’t happen automatically, and it will send a new one if it’s still having trouble.

    Thread Starter tab7x

    (@tab7x)

    Hi,
    every day Google says Googlebot can’t access your site.
    In robots-testing-tools appears a popup that inform that google cannot access my site. In this page the robots textarea is empty!
    So, if I check the robots availability all is ok.

    Moderator James Huff

    (@macmanx)

    In this case, I recommend asking over on Google’s support forums then.

    As far as I can tell, your robots.txt file is perfectly accessible.

Viewing 7 replies - 1 through 7 (of 7 total)
  • The topic ‘Google cannot access my robots.txt!’ is closed to new replies.