My site got de-indexed

Hey @sarfrazk9,

I just had a look at your website, saw that you downgraded to the previous version like you said and then checked your robots.txt file.

You still have a disallow / rule in your robots.txt file that is effectively blocking access to every user agent, including search engines. That means the problem wasn’t introduced by the new version of the plugin but was simply carried over when you updated. It’s a pre-existing rule.

I would recommend you to remove that rule immediately so that search engines can start indexing your website again. There’s no way the plugin added that rule on its own so I would make sure that no other plugins are hooking into the robots.txt file.

– Arnaud

The older version which Google webmaster saw on 27th Feb is this:

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /

Sitemap: https://www.xtremegaminerd.com/sitemap.xml
Sitemap: https://www.xtremegaminerd.com/sitemap.rss

The newer version is this

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.xtremegaminerd.com/sitemap.xml

As you can see the previous version was when the plugin was updated and now the “Disallow: /” is no longer there which was blocking the URLs from being indexed.

I hope it should be fine within a few days now but the problem is definitely in the plugin as it is no coincidence that the robots.txt changed just after updating to the newer version.

@sarfrazk9 You’re correct. I discussed this with a colleague and found that I was in fact looking at a cached version of your site’s robots.txt, which is why I still saw the disallow rule.

After reviewing our code, I’m still fairly certain this was not caused by our plugin. We did make one significant change though in the new version and that is that we now hook into the official robots.txt filter in WordPress instead of overriding it completely ourself (which is what we did previously). That means other plugins and even your theme can also hook into it.

If you happened to check the robots.txt rules in the new version of our plugin, did you see any rules there? And if not, would you mind sharing a list of your active plugins and theme with us so we can take a look and see if we can reproduce the issue?

– Arnaud

After rolling back to the previous version and then again updating to the newer version. It is confirmed that it is indeed the AIOSEO plugin that added this rule. I just tried and observed this now by updating the plugin and the rule was added automatically as my robots.txt was enabled in the feature manager. My site isn’t able to recover. 70% of my earnings lost this month and google webmaster is not allowing my sitemap to be indexed even though I have changed the robots.txt 15 days ago. Good damage is done to the site??.

@sarfrazk9 as I explained earlier, we now use the official robots.txt filter hook, which allows other plugins or your theme to hook into it as well. The rule is not coming from AIOSEO – it’s being very likely added by another plugin or theme. Feel free to contact us here (click the “Complete a Form” button) if you’d like us to troubleshoot which plugin or theme is adding it – https://aioseo.com/contact/. Either way, I’m sorry to hear about the damage that has been caused to your website’s traffic and business.

– Arnaud