Robots.txt problem Need help

  • Hi,
    I am new at WordPress. My site is working fine from last few days and Google crawls all my posts but now i got red mark on robots.txt in webmaster and my last few posts doesn’t crawl. I also check privacy and select “Allow search engines to index this site.” I have also installed Sitemap.xml plugin and select make virtual robots.txt. Now i have installed another Special plugin for Robots.txt named as “WP Robots Txt”. When i open mysite.com/robots.txt it says

    # Added by SEO Ultimate’s Link Mask Generator module
    User-agent: *
    Disallow: /go/
    # End Link Mask Generator output

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Sitemap: https://www.mysite.com/sitemap.xml.gz

    I want Google to crawl my site what should i do now.

Viewing 8 replies - 1 through 8 (of 8 total)

  • Go here for Google’s guidelines.

    Thread Starter noumanniazi

    (@noumanniazi)

    Thanks for response..
    I just want to know that my site is enable for crawling or not. In case of above Google can crawl my site???…

    Read through the link I provided. Various options are discussed there. Select the one you want. For instance, to block the entire site, use a forward slash (Disallow: /). Like this see other options.

    Thread Starter noumanniazi

    (@noumanniazi)

    According to your link my site is crawling by Google because i only select admin and includes in robots.txt. But somehow my site isn’t crawling by Google what should i do.Now some of my recent posts are displaying in Google and some are not.

    Use Webmaster Tools and Google Analytics to know what is really happening to your site and which posts/ pages are indexed/ crawled by Google. You will get all reports, answers, etc. there. Forget what others say about it.

    Thread Starter noumanniazi

    (@noumanniazi)

    I tried but and now i am conform that robot.txt file is fine but still getting red mark on Robots.txt fetch in Webmaster/tools…….

    It takes time.

    Thread Starter noumanniazi

    (@noumanniazi)

    Thanks for response.

Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘Robots.txt problem Need help’ is closed to new replies.