Failed: Robots.txt unreachable

in my website https://www.noidanutrition.com i have installed rankmath plugin and rankmath created a site map automatically but when I add sitemap in Google search console it say site map not found showing zero link. Please help me as soon as possible.

Hello @pk5959669,

It seems that you have mistyped your website URL as it is not opening.

Meanwhile, please ensure that you have submitted your sitemap index only in your Google Search Console:
https://rankmath.com/kb/submit-sitemap-to-google/

Also, the zero discovered URLs through the sitemap means Google didn’t use the sitemap to find the URLs. It mostly happens when you have good internal linking or use the Instant Indexing plugin.

Here’s a link for more information:
https://rankmath.com/kb/zero-discovered-urls-through-sitemap/

Hope that helps.

Halo,, saya juga baru saja mengalami hal ini, apakah sudah ada cara untuk memperbaikinya?

masalah yg saya alami: robots.txt tidak dapat dijangkau

tidak bisa login ke dashboard wordpress, tapi masih bisa mengunjungi situs secara langsung, ini sitis saya https://www.gerbang.or.id lihat saja tidak masalah. tapi tidak bisa login dashboard dan robot txt error.

Hello @cakmuhai,

Thank you for reaching out.

Please refer to this guide to address your issue accessing the dashboard: https://themeisle.com/blog/cant-access-wp-admin/

Checking your website’s robots.txt, we can confirm that it is loaded and accessible by Google:

https://i.rankmath.com/i/PxDVgH
https://i.rankmath.com/i/iS3n90

Looking forward to helping you.

Reviving this because I was having this issue and I was trying to find the answer here. Everything is fine on host’s side.

Site was not indexing. GSC was giving me an error with the robots.txt being unreachable. It also couldn’t fetch my sitemap.

I also had Really Simple SSL installed. Go to that plugin > Settings > Hardening > Uncheck “Disable the Built In File Editors”

Go back to Rank Math > Ensure your sitemap is available. Robots.txt should be writable.

Purge site cache (I use W3 Total Cache)

I can now request indexing of my URLs, and re-submitted the sitemap to Google.

(In my case, GSC has an error where if you submit the sitemap originally when this robots.txt issue is happening, it will say “cant fetch” instead of pending… Just TEST the sitemap URL [sitemap shouldnt be indexable, so test instead of request indexing.] Sitemap should be crawlable.)

• This reply was modified 9 months, 2 weeks ago by deciduouslypressed.
• This reply was modified 9 months, 2 weeks ago by deciduouslypressed. Reason: added additional information on gsc

Hello @deciduouslypressed,

Without knowing the website address is impossible to check whether or not that is being blocked.

You can test that with the following website: https://httpstatus.io

Make sure that you set the user agent in the dropdown to Googlebot and check the URLs of the sitemap and robots file.

Don’t hesitate to get in touch if you have any other questions.