robots.txt blocking crawlers

  • I tried to test my website with Google Mobile-Friendly Test and it returned an error that the bot is being blocked by robots.txt. I changed the contents of my robots.txt, but it changed back automatically, and I know it’s wordpress that’s changing it because there’s a “wp-sitemap.xml” related line added to the file.
    I’ve checked if it was Wordfence security and WordPress Zerospam plugins, but I couldn’t find anything related to robots,txt, and I also checked Cloudflare and my VPS’ CFS Firewall, I couldn’t find anything also. Also, when I try to generate a sitemap using an online tool, it says my site is unreachable:

    Error: URL not available (HTTP 502 ERR_CONNECTION_REFUSED)

    Can anyone check my website to see if I’m missing something?

    • This topic was modified 3 years, 11 months ago by Yui. Reason: redundant link(s) deleted

    The page I need help with: [log in to see the link]

Viewing 7 replies - 1 through 7 (of 7 total)
  • Moderator Yui

    (@fierevere)

    永子

    I dont see indexing problems on your site.

    502 can be temporary hosting problem on high load or too many request received beyond the count the server can handle.

    Your robots.txt permits indexing and defines sitemap url, sitemap works.

    Thread Starter lucrebem

    (@lucrebem)

    I tried this tool https://www.browseo.net and it couldn’t connect to the site too. But I used another website and it worked. If not robots.txt, then something else is preventing crawlers from accessing the website.

    Moderator Yui

    (@fierevere)

    永子

    You may ask your hosting support if they are blocking access from certain networks.

    You might also try pausing Cloudflare. That will let you know if CF is blocking good crawlers.

    Thread Starter lucrebem

    (@lucrebem)

    I removed CloudFlare DNS and changed back to the server’s DNS, the problem is now gone, but now I have to deal with attackers taking down CSF all day. Any suggestions of any good Firewall I could install to stop DDoS attacks?

    Jeff Starr’s 7G htaccess firewall is free and works well for me – though not specifically for DDOS https://perishablepress.com/7g-firewall/ . Maybe you gave up on CF too quickly? CF provides lotsa benefits including decent DDOS protection even on the free tier. It’s default configuration should not block anything from Google.

    If you try CF again and if it gives you grief, you might try asking at https://community.cloudflare.com/

Viewing 7 replies - 1 through 7 (of 7 total)
  • The topic ‘robots.txt blocking crawlers’ is closed to new replies.