Your robots.txt file is missing or unavailable.

Hi @ibiza69,

Thank you for reaching out and bringing this issue to our attention.

I have investigated the issue with the SEO Analyzer, and it seems that something on your site or server is blocking our analyzer when it tries to access the robots.txt file.

Our analyzer is encountering a “Please wait while your request is being verified…” prompt on the robots.txt page, which is causing it to display the message that your robots.txt file is missing or unavailable.

As seen here-
https://a.supportally.com/i/ppFQIN

You have correctly mentioned the IPs to whitelist, but it appears they are not being whitelisted correctly or by the exact service responsible for this verification prompt.

Since you mentioned that this issue has only started recently, could you please check if any new security services have been activated or if any new security features have been enabled externally or through your host settings? These new configurations might be causing our analyzer to be blocked.

Please let us know if you find any new security settings or if there are any changes that might be affecting the analyzer. We can then work together to ensure the necessary adjustments are made to resolve this issue.

Looking forward to your response!

@shivamtyagi thanks for answering, we have checked in our security plugins: Bullet Proof Security for the .htaccess file and iThemes Security (no rebranded as Solid Security) and in both AIOSEO is whitelisted, we do not have either Cache Plugins, we can′t use them because incompatiblity with our main plugins. The only problem being faced with all our configurations and plugins its this weird thing with robots.txt file.

If we ask our server admins (hosting), what would have to ask them for?, as we are confussed. Thank you for your time.

Hi @ibiza69,

Thank you for your prompt response and for checking your security plugins.

Since the issue seems to persist despite whitelisting our IPs in Bullet Proof Security and iThemes Security, it’s possible that the blocking is happening at the server level.

When you contact your server admins (hosting), please ask them to:

1. Verify IP Whitelisting: Ensure that the IP addresses 104.236.26.134 and 159.89.243.35 are whitelisted correctly in all security layers, including firewall and server-level security settings.
2. Check for Server-Level Security Measures: Look for any server-level security measures that might be causing this issue. Specifically, ask if there are any WAF (Web Application Firewall) rules, DDoS protection mechanisms, or rate limiting settings that might be triggering the “Please wait while your request is being verified…” prompt.
3. Review Logs: Review server logs to identify any entries that show the blocking of our analyzer IPs. This might give more insights into what is causing the blockage.
4. Temporarily Disable Security Measures: If possible, temporarily disable certain security measures to see if the issue persists. This can help pinpoint if a specific security setting is causing the problem.
5. Check .htaccess and Server Configuration: Ensure there are no rules in the .htaccess file or server configuration that might be blocking access to the robots.txt file for our analyzer.

Once you have this information, please let us know the outcome so we can assist you further. We are committed to resolving this issue and ensuring that your SEO setup is functioning correctly.

Thank you for your cooperation and patience.

Good morning @shivamtyagi, thanks for the help and recommendations, our server admins whitelisted your IPs and now robots.txt file is loading great in all our domains.

Thank you for your time and have a very nice weekend!