Your robots.txt file is missing or unavailable.

  • Resolved sayn0thing

    (@sayn0thing)


    9 months, 3 weeks ago

    Your robots.txt file is missing or unavailable. No matter what I try I can’t get rid of the error, I’ve tried resetting the robots.txt file through AIOSeo, but that didn’t work. I’m running WPS Hide Login not sure if that’s causing an issue. But I’ve tried uninstalling the app & it didn’t make a difference. Enable custom robots is off. Clicking on “open robots.txt” opens the file… so it exists

Viewing 8 replies - 1 through 8 (of 8 total)
  • Plugin Support Shivam Tyagi

    (@shivamtyagi)

    Hi @sayn0thing,

    Could you please clarify where exactly you are seeing the error message regarding the robots.txt file being missing or unavailable? Additionally, could you provide the URL of your site so that I can take a look?

    Thread Starter sayn0thing

    (@sayn0thing)

    Thanks for getting back to me, under Advanced SEO, there is an error saying “Your robots.txt file is missing or unavailable.” Not sure how to force a recheck so it can see it.

    The custom Robot text has:

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Sitemap: https://mindbomb.co.za/sitemap.xml
    Sitemap: https://mindbomb.co.za/sitemap.rss

    The website is https://mindbomb.co.za/ – as you can see from the sitemap ??

    Thread Starter sayn0thing

    (@sayn0thing)

    Hi, what is the status on this?

    Plugin Support Shivam Tyagi

    (@shivamtyagi)

    Hi @sayn0thing,

    Thank you for providing additional details regarding the error message you’re encountering in the Advanced SEO section. Based on your description, it sounds like you’re referring to our SEO Analysis tool.

    I’ve taken the liberty to run a test on your site using our SEO Analyzer, and it appears that access is being blocked by the Imunify360 firewall on your server. This blockage is preventing our SEO Analyzer from properly checking your robots.txt file, which is likely causing the error message you’re seeing.

    Here’s what we’re getting for your robots.txt file: https://a.supportally.com/i/85H2HI

    To resolve this issue, could you please ensure that our IPs are whitelisted on your server and firewalls? This step is crucial for allowing our SEO Analyzer to access your site without being blocked.

    You can find detailed instructions on how to whitelist our IPs in our documentation here:
    https://aioseo.com/docs/seo-analysis-unable-to-connect-to-your-site/

    Whitelisting our IPs should allow the SEO Analyzer to connect to your site and correctly check your robots.txt file, eliminating the message you’re currently experiencing.

    If you’re unsure how to proceed with whitelisting or if you encounter any difficulties along the way, we recommend reaching out to your hosting provider, as they’ll be able to assist you in making the necessary adjustments.

    Rest assured, your robots.txt file is present and appears to be correctly configured from an SEO perspective.

    Please let me know once you’ve had a chance to whitelist our IPs or if you have any further questions.

    Thread Starter sayn0thing

    (@sayn0thing)

    My hosting provider said they have whitelisted the IPs:

    I have added a rule within the .htaccess to allow the IPs of:

    ===========
    104.236.26.134
    159.89.243.35
    ===========

    Please proceed to test the plugin once more with this rule active.

    Thread Starter sayn0thing

    (@sayn0thing)

    have you checked it again?

    Plugin Support Shivam Tyagi

    (@shivamtyagi)

    Hi @sayn0thing,

    I tested your site with our SEO Analyzer again and it seems to be working fine now. I’m not seeing that message for robots.txt anymore.

    Could you please go to?All in One SEO > SEO Analysis?and click on the?Refresh Results?button and let me know if you’re still facing the same issue?

    Thread Starter sayn0thing

    (@sayn0thing)

    Excellent! It seems to be working now. Thanks ??

Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘Your robots.txt file is missing or unavailable.’ is closed to new replies.