Robots.txt problem
-
I have several wordpress installations working fine – this situation is unique. I’ve searched the forums and found similar issues and recommendations, however none seem to help.
Google says there is an issue with my robots.txt and the error link points directly to my front page. This is not a cached version of the robots.txt file… or at least Google says it crawled this morning. I generated my sitemap with the Google sitemap plugin by Best Web Soft plugin. It looks accurate upon review.
Reviewed all plugins and removed one plugin that hides the title on my front page… no help.
Checked my front page for any meta tags that might be blocking progress – nothing.
The robots.txt now shows the following.
User-agent: * Disallow: /cgi-bin/ Sitemap: https://mysite.com/sitemap.xml
I’d like to button up the wp-admin and plugins directories but for now all is open.
Checked the site at https://tools.seobook.com/robots-txt/analyzer/
and it says all pages are open.I checked my database and made sure the robots entry data contained the correct amount of characters since wordpress has been moved from a subfolder to the root. All paths in the database are correct.
Reinstalled WordPress
Any advice regarding something I might be overlooking would be greatly appreciated. Google refuses to index my front page.
Thank you.
- The topic ‘Robots.txt problem’ is closed to new replies.