What is the normal Robots.txt file should be and will search engine crawl my po?

  • sujoydhar

    (@sujoydhar)


    13 years ago

    Hi I am new to this WordPress and don’t know what would be the normal Robots.txt file would be ?

    My website is https://www.wideinfo.org and I have googled the search and add something to my Robots.txt the content is given below

    User-agent: *
Disallow: /wp-content/
Disallow: /wp-icludes/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /archives/
Disallow: /category/
Disallow: /tag/*
Disallow: /tag/
Disallow: /wp-*
Disallow: /login/
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.php$

Sitemap: https://www.wideinfo.org/feed

    Now will the search engine Crawl the Post pages ?

    At present what are the effects in search engine by the following above mentioned content in Robots.txt

Viewing 1 replies (of 1 total)
Viewing 1 replies (of 1 total)
  • The topic ‘What is the normal Robots.txt file should be and will search engine crawl my po?’ is closed to new replies.