How to tell googlebot to crawl just one URL with .html
-
Hello
I have a big problem in webmaster tools, 404 increases so fast, now I got over 1000, when I check for errors I see every page googlebot catches URLs without .html and that creates 404 error.
I tried to see source of error, but I see, for example this one: https://ermagazin.com/najgora-nuklearna-katastrofa-u-americkoj-povijesti-za-koju-nikad-niste-culihas 3 sources and all 3 are corect link, one of them is https://ermagazin.com/najgora-nuklearna-katastrofa-u-americkoj-povijesti-za-koju-nikad-niste-culi.html which should be catched instead the first one without .html
Now, what do I need to add in robots.txt so googlebot crawl just one url with .html
My robots.txt is:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
Disallow: /readme.htmlSitemap: https://ermagazin.com/sitemap_index.xml
https://ermagazin.com/post-sitemap.xml 2016-02-11 08:57
https://ermagazin.com/page-sitemap.xml 2016-01-14 14:45
https://ermagazin.com/category-sitemap.xml 2016-02-11 08:57
https://ermagazin.com/post_tag-sitemap1.xml 2016-02-11 08:57
https://ermagazin.com/post_tag-sitemap2.xml 2016-02-11 08:57Any suggestion please, this drives my crazy ??
- The topic ‘How to tell googlebot to crawl just one URL with .html’ is closed to new replies.