robots.txt problem

  • Is there anything in the below robots.txt contents that would stop Googlebot from crawling and indexing https://www.insuranceclaimhelp.org ? This was the message posted by Wayback
    https://wayback.archive.org/web/*/https://www.insuranceclaimhelp.org/

    (you will need to copy paste the entire url in your address bar. It was apparently not all highlighted above?)

    robots.txt :

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/
    Disallow: /trackback/
    Disallow: /images
    Disallow: /feed/
    Disallow: /comments/
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /comments
    Disallow: /peoples-choice
    Disallow: /*?*
    Disallow: /*?
    Disallow: /*?
    Disallow: /*/*rate
    Disallow: /*/*add
    Disallow: /aboutadj.asp
    Disallow: /?*

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*
    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*
    # Internet Archiver Wayback Machine
    User-agent: ia_archiver
    Disallow: /
    # digg mirror
    User-agent: duggmirror
    Disallow: /

Viewing 3 replies - 1 through 3 (of 3 total)

  • You don’t need all that stuff.

    The only things I’ve got in my robots.txt is this

    User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
    Thread Starter jetskiron

    (@jetskiron)

    Thanks Peter,

    What kinds of things would be in the wp-includes that robots should not see?

    Not sure to be honest. It’s just a snippet I picked up somewhere.

    The point being, you don’t really need robots.txt in WordPress.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘robots.txt problem’ is closed to new replies.