Sitemaps and Google Search Console

Hi @duts2020 this is not something I have seen happen before. I wonder if there is some caching problem or conflict. Can you share a link to your site(map) so I can verify the response headers?

Hi RavaH,
Here is the link: https://arbalettv.com/sitemap.xml

Hello again RavaH,
Have you had a chance to check my sitemap? I don’t see any issues but the problem inside Google Search Console still persists.

Hi, looking at your response headers, I only notice batcache active. I do not think that should cause the issue but you could try submitting your sitemap with an added request parameter like /sitempa.xml?nocache to completely avoid batcache… Let me know ??

Hi RavanH,
I have tried what you have suggested and I can now confirm that it didn’t have any effect. Sitemaps are still disappearing.

I have also been testing for the past several weeks Jetpack’s sitemaps along with sitemaps generated by your plugin. I have disabled Jetpack’s sitemaps but they are still showing up when I test pages using URL inspection tool. They were never dropped. It is a very interesting technical issue.

Hi RavanH,
Just letting you know that I have tested another plugin along with this one. That plugin’s sitemaps were not dropped as well. So we are talking about 2 other sitemap generators that I have tested and in each case those sitemaps were not dropped and they haven’t disappeared after certain amount of time.
I don’t know why but your sitemaps are disappearing while sitemaps created by other plugins aren’t. This is something to think about and I wanted to share my experience. Some issues started to come up after Google’s end of June update that was supposed to target only spam. Not sure if it’s somehow connected but I wanted to let you know that I experienced crawling/indexing issues and then disappearance of your sitemaps when individual pages were checked using URL inspection tool. Everything ran smoothly before Google’s update.

Hi RavanH,
I would appreciate if you can reply after reading my two latest post replies. Thank you.

Hi @duts2020 (just back from short vacation) thank you for your detailed description and reporting the issue. I am completely in the dark however about the cause of the issue.

You wrote:

5-6 hours later and the URL inspection tool is showing “URL is unknown to Google” and “N/A” next to sitemaps in the Discovery section.

Do you mean that in Search Console, you click on Sitemaps, then click on the Sitemap Index (sitemap.xml) listed under “Submitted sitemaps” you get your individual sitemaps listed there with “N/A” next to some of them? If so, which ones and are they always the same ones even after refreshing?

Hi RavanH,
Sitemaps are listed in the Search Console. When the sitemaps are updated everything is working fine for 5-6 hours. After that, when you run the URL inspection Tool for any individual page the discovery section lists N/A for sitemaps and there are no sitemaps listed there. They were listed there for 5-6 hours and then they are simply gone. When you go to sitemap section of the Search Console and re-submit the sitemaps and then re-run the URL inspection tool again then the sitemaps would appear again and stay this way for the next 5-6 hours after which they would be dropped again. I have tested 3 other sitemap plugins and in each case the sitemaps are always shown when the URL tool is run. At the same time I can confirm that everything was working fine up until July and the issue is isolated to Google Search console.

When you go to “Coverage” (under the Index section on the left) you will see a select field on the top under the Coverage title, saying “All known pages”. If you click the down arrow, do you see and option “Filter to sitemaps” under which should be listed your main sitemap.xml index. Are there any errors shown then?

To explain: the fact that your sitemaps do not appear in the URL Inspection tool is because they are marked “noindex”. This is by design, to prevent the sitemaps themselves showing up in the search results.

RavanH,
I never said that I was checking the sitemaps using URL Inspection tool. When you run a check for any page (excluding sitemaps of course) there are 3 sections on that page in the coverage area namely DISCOVERY, CRAWL and INDEXING. In the discovery section there are always sitemaps listed in which that particular page is mentioned. So when I talked to you I meant the discovery section. Your sitemaps are listed there for 5-6 hours and then they are dropped off from the discovery section when you re-run the inspection tool. It would have been easier if I could send you a screenshot so we can understand each other better.

To answer your first question. There are no errors listed there that could have affected the issue I have described.

Ahhh, it’s getting clearer for me now…

To make sure: the posts themselves remain indexed, just the mention of the sitemap index is dropped?

And via Coverage > Filter to sitemap > sitemap.xml? Click the “Valid” tab, then follow the link “Submitted and indexed” under Details to open the list of pages/posts… Can you find the affected posts (where the sitemap dropped off the Discovery info) there or are they not present in the list?

RavanH,
As I have mentioned before I have been using the second set of sitemaps generated via another plugin. I have had indexing issues/crawling issues and I couldn’t find the reason. Then I found out that your sitemaps were dropped and the discovery section was empty. I have added another set of sitemaps via different plugin and they are not dropping. The issues with indexing and crawling have cleared once I added additional sitemaps. Now I am trying to figure out why your sitemaps are dropping in the first place because until the end of July everything worked more or less normally. I have schema markup for video pages and I have had issues when Google failed to show my thumbnail and was showing my site’s logo instead. I researched all the possible reasons and found that those pages were re-crawled and re-indexed when the sitemaps were dropped and the discovery section was empty. This is very strange issue. I have added a second set of sitemaps and re-crawled and re-indexed the pages which are not showing the correct schema thumbnail. All these attempts to figure out the issue have led me to the fact that your sitemaps are dropping off from the discovery section for no apparent reason and when this is happening the pages are experiencing issues with crawling and indexing.

Since the other plugin’s sitemaps are not being dropped and I have tested at least 3 other plugins so far (including Jetpack’s sitemaps) then I guess it’s reasonable to suggest that there is some sort of conflict between Google and sitemaps created by your plugin.The URL inspection tool is supposed to remember the information read at the time of the test.

• This reply was modified 3 years, 3 months ago by duts2020.

Sadly, I have absolutely NO idea why this could be happening, or how it even could be possible that a sitemap that otherwise shows no errors could cause this. No other reports about this have reached me but I’ll definitely will keep a close eye on this.

Thank you again for the detailed information and your extensive testing!