seeing in robots “content=”noindex, follow, …”. Why ‘noindex’?

@toad78 We’re sorry to hear that you are experiencing this issue.

There are several ways that you can set a page to noindex. We would recommend checking to make sure the robots meta tag is set to index for the page/post in Yoast SEO in each of the following ways.

Setting a single page/post

You can check this setting by performing the following actions:

? In WordPress, go to the Edit screen for a page/post
? Scroll down to the Yoast Metabox
? Click on the ‘Advanced’ area
? For ‘Allow search engines to show this Page in search results?’ make sure this is set to ‘Yes’
* Note (setting this value to Yes will set meta robots to index and setting No would set meta robots to noindex)

Setting a content type or taxonomy

You can set all pages/posts and taxonomies to index/noindex by performing the following actions:

? In WordPress, go to SEO>Search Appearance>Content Types
? For Pages set ‘Show Pages in search results’ to ‘Yes’
? Click ‘Save Changes’

We also have a guide available on how to set a page/post’s meta robots tag to index/noindex here: https://yoast.com/help/how-do-i-noindex-urls/

Hello, @devnihil :

The point I’m making is that in the source code ALL of the pages/posts are posting “content=’noindex, follow,’ …”, which is wrong. All of the Pages/Posts are all set to be indexed/followed but the source code says the opposite. So even though I would edit each and every Page/Post to ‘force’ it to be folowed/indexed, it doesn’t do any good.

The same goes for going to Yoast SEO/Content Types > Pages/Posts > Show… > Save. While everything is set to ‘Yes’, nothing changes.

@toad78 If you are seeing all the sites pages as being noindex, we would also recommend checking the global setting for the site.

You can check this in the following way:

? In WordPress go to Settings>Reading
? Make sure for ‘Search Engine Visibility’ that ‘Discourage search engines from indexing this site’ is NOT checked
? Click ‘Save Changes’

Nothing was checked on “Discourage search engines…”

Hi,

Since you’ve checked all relevant settings within the Yoast SEO plugin and it is still showing with a ‘noindex’ meta robots tag in the page source, we would like to check if there are any plugin or theme conflicts.

The fastest way to do this is to?deactivate all non-Yoast plugins and switch to a standard theme?like?Twenty Nineteen.?

Test this on your development or staging site, if you have one. If not, we recommend using the?Health Check & Troubleshooting?plugin. This plugin allows you to run a conflict check?without affecting normal visitors to your site.

As we can imagine that you’re unfamiliar with checking for conflicts, we’d like to point you to a step-by-step guide that will walk you through the process:?How to check for plugin conflicts.

Deactiving all plugins prior to switching to a theme results in the correct meta content:
<meta name=’robots’ content=’index,follow’ />. After I enable ALL of them, except for Yoast, they still post the correct meta content. It’s only after I enable Yoast that the ‘noindex’ appears.

With Yoast disabled (but my other plugins enabled), I changed to the “Twenty Twenty” theme and still see the “<meta name=’robots’ content=’index,follow’ />”. After enabling Yoast (standard) on the “Twenty Twenty” theme: “<meta name=”robots” content=”noindex, follow,…>”

@toad78 Thanks for your reply and for the additional information.

Can you please confirm whether your site uses both posts and pages? If it does, can you check to see whether the noindextag is output for both?

Also, can you please try the following steps to see if it resolves the problem with thenoindex tag being output on your site?:

? In WordPress, go to SEO>Search Appearance
? Click on ‘Content Types’
? For ‘Show Pages in search results?’ set this to ‘No’
? Click ‘Save changes’
? Then for ‘Show Pages in search results?’ set this back to ‘Yes’
? Click ‘Save changes’

If the issue is occurring on both pages and posts, can you also try the same actions for the ‘Posts’ content type as well?

Hello, @devnihil .
1. Yes, the site uses both Posts/Pages.
2. ALL Posts/Pages have the ‘noindex’.
3. ‘Show Pages in search results?’:
a. after Setting to ‘No’ and refreshing the Source Code for both Pages/Posts:
<meta name=”robots” content=”noindex, follow” />
b. after Setting to ‘Yes’: same

@toad78

Could you please show us some example URLs where the issue occurs so that we can take a look at them?

Also, we had a new update released yesterday, v15.7. If you update, does the issue resolve?

I would need to reply with the URLs privately and not publicly.

The Yoast was already updated prior to yesterday’s last comment.

@toad78 If you need to send us links privately we are unable to do so here since this is a public forum. However, you may consider purchasing Yoast SEO Premium that will grant you access to our premium support where our support engineers can investigate the further for you and offer support through emails privately.

Temporary URL: https://montestaging.wpengine.com/

Thank you for providing the cloned version of live site. While we investigate the issue on our end, we request you the following.

– Temporarily remove the Disallow: / rule from the robots.txt file on your staging site. We also understand you don’t want search engines to crawl your staging site. Since your concern is with noindex robot not working as expected on site pages/posts and we need to debug it treating like a live site.

Recommended robots.txt code:

User-Agent: *
Disallow: /wp_admin/
Allow: /wp-admin/admin-ajax.php

– Deactivate all non-Yoast plugins and switch to the standard theme like Twenty Twenty-One.

– Make sure no posts and pages are specifically set to noindex from edit page/page.

– Make sure all content types settings are set to Yes for search engine visibility in SEO > Search Appearancee> Content Types.

We will investigate the issue based on the above config.

– Temporarily remove the Disallow: / rule from the robots.txt file on your staging site.

Done

– Deactivate all non-Yoast plugins and switch to the standard theme like Twenty Twenty-One.

Done.

– Make sure no posts and pages are specifically set to noindex from edit page/page.

For now, let’s just work on the Home page, which is already set to be followed and indexed by default.

– Make sure all content types settings are set to Yes for search engine visibility in SEO > Search Appearancee> Content Types.

It always was.

We are sorry to hear that you’re still experiencing the noindex issue on your site. Due to the nature of the issue, we’ll need to take a look at your setup. However, we are unable to do it here as this is a public forum. So, if you have a Yoast SEO Premium subscription or if you’re willing to buy one, you’ll have access to our premium support where we can investigate the issue directly on your website privately.

If you don’t wish to have the Yoast SEO Premium, that’s completely fine. In this case, other community members might be able to help you with this as we are now out of clue without taking a look at your setup.

Thanks for your understanding!