ROBOT.TXT Not Allowing Crawling of my Site

Viewing 5 replies - 1 through 5 (of 5 total)
  • Thread Starter jbendiner

    (@jbendiner)

    Sorry, the link should read:

    https://www.technologix.ca/robots.txt

    thank you.
    Jerry

    Try just using without the WWW. at Google and Bing.

    The www is a redirect to //technologix.ca/

    Check each about this issue.

    Thread Starter jbendiner

    (@jbendiner)

    Thank you for trying to help!

    Not sure I understand your WWW suggestion, sorry.

    Both Google and Bing are claiming that the pages can’t be crawled, when I search using google and Bing. Something is not allowing them to crawl my site yet the robots.txt file seems sound. My problem is not with google or bing. It is somewhere between wordpress and my robots and not sure where to look.

    Both Google and Bing are claiming that the pages can’t be crawled

    Where are you seeing this?

    Thread Starter jbendiner

    (@jbendiner)

    If you go to bing.com, for example, and in their search box you type my url, it will display the following result:

    https://www.technologix.ca
    We would like to show you a description here but the site won’t allow us.

    I also checked my wordpress reading settings and I noticed that the
    “Discourage search engines from indexing this site” was checked which would obviously cause this problem. I have since unchecked it.

    I wonder how long it would take for Google and Bing to register this change and start crawling again…

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘ROBOT.TXT Not Allowing Crawling of my Site’ is closed to new replies.