Missing from search if primary category is missing

The problem must be something else. Relevanssi completely ignores the Yoast SEO primary category setting in any case: for Relevanssi, it just doesn’t exist, so it doesn’t and cannot have any effect on the search.

So please explain in more detail what’s happening and how exactly the search is not working. Are you talking about searching by category name, or using some kind of category filtering?

Hi Mikko,

well, it was just a coincidence, we were searching for some post in the past (we have around 10k posts) and two of them simply were not shown in the regular search at all. After we updated them with a primary category, it just immediately started to be visible in the search.

I don’t know, maybe there is some filter which is excluding them? But we don’t use any include or exclude categories, nothing is checked. So I suppose that everything should be also indexed and visible on the site while searching for it.

It is hard to find out now which other posts are buggy because I don’t know which ones are similar. Maybe it’s not just about the primary category, maybe it’s somewhere else.

Is it possible to download or filter from the database table all the posts the plugin has indexed to be able to compare it with all of our articles? Or if you have other idea what might be wrong here, it would be great.

Thank you.

Well, for starters you could just rebuild the whole index, in case there are some problems with the index. Updating the post reindexes it, so if it was missing for any reason, that would fix it. Thus, rebuilding the whole index would probably help.

The index is stored in the wp_relevanssi database table. If you want to see a list of posts not indexed by Relevanssi, you can do this in your database:

SELECT ID, post_title FROM wp_posts WHERE ID NOT IN (SELECT DISTINCT(doc) FROM wp_relevanssi) AND post_status = 'publish'

This will show you the ID number and the post title for all the published posts that are not in the Relevanssi index.

Hm, I don’t know why but it looks like the indexed pages were simply deleted from the index recently. Was there any update from your side or under which occasion this can happen? Deactivating should not delete the database table but I am not even aware of any deactivation…

Nothing in particular in Relevanssi. Could be caused by another plugin: are these posts perhaps marked as noindex in a SEO plugin like Yoast SEO? That will remove them from Relevanssi index as well.

No, but yoast was having a HUGE update on his indexing features and had to be reindexed with them as well.

The problem is somewhere else. I just indexed 9000 posts in your plugin. But when I go to the plugin to check it then I can see this

State of the index
1728 documents in the index.

580724 terms in the index.
954 is the lowest post ID indexed.

I am pretty sure that I can run Index unindexed posts again and again 9k will be running and indexing and in the end nothing will happen… ?? ??

I just checked your command in the database and it is the same as before the indexation. Over 11k of rows.

So I don’t know now how to index our posts or what is the problem here…

That’s weird, but I do have an idea what’s up.

Would you by chance happen to use some kind of shortcode that includes a post loop? Something that shows posts inside posts? Those kinds of shortcodes can short circuit the indexing if they change the global $post. That would lead to something exactly like this: Relevanssi thinks it’s indexing all posts, but because the global $post is changed during the indexing, not all posts get indexed (and lots of post content gets indexed for the wrong post).

If you disable shortcode expansion and index again, does everything get indexed? If so, this is probably the reason. The fix is to adjust the shortcode so that it doesn’t use WP_Query and doesn’t change the global $post, but instead uses get_posts() and doesn’t touch the global $post. There’s no reason why a shortcode should do a loop and have anything to do with the global $post like that, that’s just sloppy programming that’s begging for trouble.

We use Huber theme https://themeforest.net/item/huber-multipurpose-review-theme/19858260

There are different post types, some are game hubs, some are posts, some are reviews etc. But we do not use anything extraordinary. Also, 1700 posts are indexed, but the rest around 8k is not. And those indexed posts are exactly the same as the others.

Also, as I said, if I update (resave) some unindexed post, it gets indexed immediately and we make no changes.

There may be some wrong connections or who knows what because we have imported 9k pages from a different database and the whole site and posts are migrated. But again, all the posts are there, visible, in the database etc. And the post has a post_type = post in the table which is the standard. Also removing from your settings to index custom posts (set to none) did not help.

I think this is something which should be analyzed on a deeper level or we need to know what your plugin is using to index them, what connections, fields etc. to check them. I don’t know…

Edit: Regarding the loop, you can check it for example here https://www.keengamer.com/games/minecraft-dungeons/ that is a post type as well, just used as a hub. And there are more posts visible as news etc. But this cannot be changed in any way. And again, as I said, all the new articles are indexable.

• This reply was modified 4 years, 10 months ago by keengamer.
• This reply was modified 4 years, 10 months ago by keengamer.

Huber is a premium theme, so that doesn’t help me, because I can’t see what’s inside.

But it sounds like those hub posts may very well be the cause of the problem. If they have a badly-built post loops, they would cause something exactly like this.

How those hub pages work? It’s a post, does it have a shortcode in it or something like that? How does it look like when you go edit a post like that?

Did you try rebuilding the index with “Expand shortcodes” disabled? If you didn’t, try that – it’ll be very useful to know if that helps.

Okey, so after disabling shortcodes, it works. But I cannot even add all custom fields. And now I have another problem. If you go for example here https://www.keengamer.com/articles/features/opinion-pieces/sonys-next-gen-silence-is-deafening-but-working/ then there is a text below the title. That is used as “subtitle” field. No matter what I do and adding it to the custom fields, it is never indexed and visible in the search. We have also “synopsis” or “summary” which is used during the post creation. See here https://imgur.com/a/SlXDq45

Any ideas now?

K.

What do you mean exactly when you say you cannot add all custom fields?

How have you set up the Relevanssi custom field indexing settings? If you set custom fields to “visible” and rebuild the index, you cannot still find posts by subtitle content?

If I choose ALL custom fields, there is the loop again and indexing takes 4 hours. If not, it takes 10 minutes and everything is indexed. And yes, even using Visible, that part is not indexed. Or at least, if I search for some text from it, it’s not found…

Hm, I don’t get it. Search for Skyrim and you get this https://www.keengamer.com/?s=skyrim Then if you go to the first post, I tried to search for

“We bring you the next list in an overview of our favorite Skyrim” which is in the body of the article, it’s not in the search.

Also “You will never guess that Elder Scrolls: Skyrim” from the subtitle does nothing.

Why is that? It should clearly get this text and put the article to the first place…?

If you disable shortcode expansion and include visible custom fields in the index, does the indexing still take a long time? I can understand what’s going on if you enable shortcode expansion and the custom fields take a long time to index, but if shortcodes are disabled and the custom fields get the indexing stuck, then I’m not sure what’s going on in there.

What you’re seeing with Skyrim is exactly what I’ve been talking about: the post loop gets Relevanssi confused and Relevanssi ends up indexing post content for the wrong post. When Relevanssi is indexing the Skyrim post, something changes the global post object and the content ends up indexed for the wrong post, hence the weird results and missing posts you’re seeing.

This is straightforward to fix: all it takes is figuring out the name of the shortcode that’s causing all this, and making sure that shortcode is disabled when Relevanssi is doing the indexing. If you can’t figure out the name of the shortcode yourself, you can ask the theme support, they should know. The hub content is something you don’t really want to index anyway; I don’t think you want to find the hubs in search, it’s better to get the actual posts, not the hubs that have the posts.

Since you’ve bought the theme and can contact the author, could you please do me a favour and send them this note:

“Huber hubs don’t work correctly with Relevanssi search plugin indexing. Looks like the hubs are using WP_Query to fetch the posts and in the process change the value of global $post, which gets Relevanssi indexing confused. Using get_posts() and not touching the global $post would be a safer way to do the hubs and would guarantee compatibility with Relevanssi and other plugins that rely on the global $post being left untouched in situation where it shouldn’t change. For more information, you may contact me at https://www.relevanssi.com/contact/ “

• This reply was modified 4 years, 10 months ago by Mikko Saari.