url restricted by robots.txt

  • When looking at Google webmaster tools the following message is displayed :
    url restricted by robots.txt

    I have as well the module WP Robot3

    How do I control those robots ? Also I have insert
    User-agent: *
    Allow: /

    at my root dir. I though that will allow crawling all my pages !
    Help I am lost…..

Viewing 1 replies (of 1 total)

  • You can very easily manually create a robots.txt file – you really don’t need a third party app to do it. Just create a manual robots.txt. Decide what directories you want to exclude, create a plain-text file called robots.txt, and upload it to your root. I usually exclude wp-admin and wp-includes as well as cgi-bin if it exists, along with any other folders I don’t want getting spidered by search engines. Your needs may be different.

    More info about robots.txt and its syntax here: https://www.robotstxt.org/

    I was not familiar with WP Robot3 – is it this? https://wprobot.net/ It seems to be some automated blog-posting tool. Does it have anything to do with robots.txt? I wasn’t sure why you mentioned it.

Viewing 1 replies (of 1 total)
  • The topic ‘url restricted by robots.txt’ is closed to new replies.