Review of my robots.txt file.

  • Can someone take a look at my robots.txt file below and tell me if there’s a problem with it — or does it simply take search engines a while to crawl and properly index new sites? I keep reading that it takes “some time” but have no idea if that means hours, days or weeks. For the last 5 days, when i do a google search on goldendaleevents.com, the result i get is: “A description for this result is not available because of this site’s robots.txt – learn more.” In spite of this, some, but not all, of my pages are starting to show up (i’m in the process of learning about/revising the metadescriptions).

    Here’s the content of the robots.txt file, which is in the root of my domain directory:

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-content/
    Disallow: /wp-content/plugins/
    Disallow: /wp-includes/

    User-agent: Googlebot
    Allow: /

    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    User-agent: Adsbot-Google
    Allow: /

    User-agent: Googlebot-Mobile
    Allow: /

  • The topic ‘Review of my robots.txt file.’ is closed to new replies.