[Plugin: Search Engine] Search Engine not following robot.txt/noindex rules

The robots.txt parser is not that advanced yet, and I was looking for your nofollow/noindex in robots tag on a few of the pages but didn’t see it, which pages have it set so I can test?

As a solution for not indexing certain URIs, you could try using the blacklist setting in the Search Index Template advanced section of the Wizard when creating a new template. If you’ve already indexed, you can go to Search Engine >> Settings >> Reset to clear out your data and start over

Basically every page/post has a print version so my whole site is being index twice; once for the regular page and once for the print version. It’s also indexing the shortcode version even though the robot text asks it not to. So, our website with 1000+ page now is trying to index 3000+ pages with 3 of the same search results for each query.

noindex, nofollow example: https://apexprd.org/home/printpage/

I’ll give the blacklists a try. Thanks-

It also might be worth trying to set your meta tag’s name to “robots” instead of “Robots”, I don’t think I had a strtolower set on that loop, I just added it to my dev copy.

Thanks, I never gave it a second thought.