Unreachable robots.txt

  • Resolved pickriff

    (@pickriff)


    10 years, 10 months ago

    iI have inherited a wordpress site, that is highly complicated and has lots of plugins.

    It looks like no crawlers are being allowed to access the site getting a 500 error. having looked at the logs it started just under a month ago.

    So far i have removed everything from.htaccess file except basic stuff, checked all plugins, deactivated plugins checked error logs with no way forward. Google and other crawlers simply refuse to crawl it. please note the frontend and admin sites are accessible with no problems.

    so i am at a loss as to what is causing it and how to fix.

    Site is https://www.berlitzmanchester.com

Viewing 6 replies - 1 through 6 (of 6 total)

  • Have you tried:

    – deactivating all plugins to see if this resolves the problem. If this works, re-activate the plugins one by one until you find the problematic plugin(s).

    – switching to the default theme to rule out any theme-specific problems.

    – resetting the plugins folder by FTP or PhpMyAdmin. Sometimes, an apparently inactive plugin can still cause problems.

    Also, try checking your site’s error logs for a more specific error message. Your hosts should be able to help you accessing your site’s error logs.

    I can’t directly see any problems. Google only has 2 pages (uploads) indexed. Bing knows a lot more but also mainly uploaded pdf’s.

    Do you have an account with google webmaster tools. That might tell you where the problem lies.

    The English home page throws an error.

    How do you know the site is throwing an error 500?

    Thread Starter pickriff

    (@pickriff)

    Ive used Screaming frog and also webmaster tools, shows a unreachable 500 error.

    however i think ive found the problem, one plugin Nextgen gallery, ive disabled it and screaming frog is no longer gettting a 500 error,

    however fetch as google in webmaster tools is still showing unreachable robots.txt

    Thread Starter pickriff

    (@pickriff)

    the error is because i have disabled the plugin that controls the homepage picture

    Google webmaster tools takes a few days to update.

    Thread Starter pickriff

    (@pickriff)

    i thought that might be the case, all ineed to do is rebuild it it without next gen gallery awesome!!! thanks for your help guys

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘Unreachable robots.txt’ is closed to new replies.