Issue with robot.txt
-
Hi All!
So I have been trying to plug the holes on my wordpress site to get rid of duel content and have had pretty good luck with my robot.txt file it has blocked a lot of unwanted stuff BUT this morning checked my site in google and some things are still leaking out :
domainname.com/blog/tag/blah/
and
domainname.com/blog/category/blah/Below is my robot.txt code:
Sitemap: https://domain.com/sitemap_index.xml
User-agent: Mediapartners-Google*
Allow: /User-agent: Googlebot-Image
Allow: /wp-content/uploads/User-agent: Adsbot-Google
Allow: /User-agent: Googlebot-Mobile
Allow: /# global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?
Disallow: /go/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: ?comments=*
Disallow: /search?
Disallow: /?p=*
Allow: /wp-content/uploads/Do I just add Disallow: /blog/category/*/* and Disallow:/blog/tag/*
Any ideas?
- The topic ‘Issue with robot.txt’ is closed to new replies.