Sitemap Problem Google Console not indexing

@tonybreadtag

We’re sorry to hear that you’re having trouble with sitemaps as per the Google Search Console report.

Taking a brief look at your website https://breadtagsagas.com/ we do see the Yoast sitemaps https://breadtagsagas.com/sitemap_index.xml are generated fine as expected.

Are you seeing any errors in Google Search Console related to sitemaps or URLs being unable to crawl or index? If so can you please provide us with more details on the issue?

• This reply was modified 3 years, 4 months ago by Suwash.

The error appears mainly unable to Fetch! And one is reading XML file as HTML.

https://breadtagsagas.com/sitemap.xml Fetch HTTP Error:
500 Site could not be read. And the same for:

https://breadtagsagas.com/post-sitemap.xml
https://breadtagsagas.com/category-sitemap.xml
https://breadtagsagas.com/page-sitemap.xml
https://breadtagsagas.com/post_tag-sitemap.xml
https://breadtagsagas.com/sitemap_index.xml

This is only in error
https://breadtagsagas.com/attachment-sitemap.xml

Thanks for providing more insight, Tony. First, we will like to clarify that your Yoast-generated XML sitemap index is at https://breadtagsagas.com/sitemap_index.xml and not https://breadtagsagas.com/sitemap.xml. Hence, we are unable to troubleshoot issues with third-party-generated sitemaps.

As you can see here, there are six individual sitemaps in the index and they all load correctly. We see no reason why Google will not be able to fetch.

Kindly follow the instructions below to remove and re-submit your sitemap:https://yoast.com/help/submit-sitemap-search-engines/. Do note, that, you only need to submit the sitemap index – /sitemap_index/.

Do let us know how it goes

Thank you for your help to date! I have erased the https://breadtagsagas/sitemap.xml from Google Console.

I had already followed the steps to re-submit my sitemap on Yoast as outlined before I contacted you.

I am hoping that from my site’s point of view the Sitemap issue has been resolved. I just need Google to recrawl and re-index, I think. (Maybe I am wrong and have no idea!)

Should I erase all the six sub-sitemap URLs from my Sitemaps on Google Console.

On the console all the Sitemaps listed say Couldn’t Fetch but for example the https://breadtagsagas/sitemap_index.xml says submitted 25 October but last read September28,2021. Does this mean I have to wait until Google looks at my site?

How Am I going to get back the thousands of university students many from developing counties who are reading my EH Carr blogs?

I have submitted some re-index requests to Google but am not a hundred percent sure that I have done this correctly.

Is there any way I can get Google to hurry this up? Do you need to gain access to my Google Console to see. I would send you screenshots but do not think I can.

Kind regards
Tony
P.s. I am tearing my hair out over this and can’t get over the foolishness of letting it drag on and not noticing.

Hi @tonybreadtag

Yes, please make sure you’ve followed the steps in this article to remove /sitemap.xml and just make sure that you only resubmit /sitemap_index.xml. You’ll have to wait and monitor for the next couple of days and see if Google decides to recrawl your sitemap and update the ‘last read date’ accordingly.

Good news and bad news!

Google for some reason crawled https://breadtagsagas.com/post_tag-sitemap2.xml yesterday and the result was Success. Hopefully, they’ll crawl more. I don’t know how to encourage them to crawl https://breadtagsagas.com/sitemap_index.xml next. Any suggestions?

About 23 September, I updated to WordPress 5.8.1 had a conflict with jetpack and had to erase it and reload it. Then broken links told me that most of my posts were broken. I dismissed all these but they were a sign of an actual problem. As well as losing my sitemap, there is probably something wrong with the bulk of my posts.

I did a test with Google I thought on https://breadtagsagas.com/what-is-history-5-eh-carr-historians-their-facts/ they responded:

While your site can be accessed via a browser, Googlebot is unable to access it. This could be due to a protection system that’s blocking unknown requests to your site. Please reach out to your hosting service provider to check what’s the underlying issue and fix it to avoid further access issues.

I’m not sure how that process is going but should try further as looking at the history post today Google says it seems OK and I’ve requested to Index it. I’ll do a few more checks on other posts and also contact my provider Siteground but I don’t think they would have changed anything around late September to cause this.

Another site suggested I input breadtagsagas.com/robots.txt to look for problems. I don’t know what this does or why, but got the response:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
This didn’t seem to be a problem to me (a naive non-expert) as I was supposed to be looking for Disallow Googlebot or * Disallow / meaning anything.

I am being stretched beyond my competence here and would appreciate any suggestions or advice. Or what I my next approach to Google should be.

Kind regards
Tony

I tried running https://breadtagsagas.com/sitemap_index.xml through Google Console’s URL Inspector and requested indexing then got the response ! under Indexing Allowed?

No: ‘noindex’ detected in ‘X-Robots-Tag’ http header

I don’t know if this indicates a ‘Rogue X-Robots-Tag’ or this is to be expected because I am trying to Index a sitemap the wrong way.

Confused

Cancel that last request!!! I tried https://breadtagsagas.com/post_tag-sitemap2.xml and got the same result. So I suppose I know the answer.

It is dangerous to let beginners loose!!!!

Hello Tony,

We have addressed your concerns below:

I don’t know how to encourage them to crawl https://breadtagsagas.com/sitemap_index.xml next. Any suggestions?
Do you mean that the other individual sitemaps were read with success but failed for the sitemap index?

Another site suggested I input breadtagsagas.com/robots.txt to look for problems. I don’t know what this does or why
The robots.txt file is correct as there are no directives in it that hinder indexing

‘noindex’ detected in ‘X-Robots-Tag’ HTTP header
The ‘noindex’ X-Robots-Tag is expected. We want Google (and other search engines) to crawl the XML Sitemaps and detect the links inside them, but we do not want the XML Sitemaps to be shown in the search results as they’re not meant to be used by regular users, just by crawlers.

In addition, we checked your sitemaps in a sitemap validator and there are no errors. See here and here.

Thank you very much!!! You have been extremely helpful in showing me the route through this difficult problem.

I have subsequently managed to get my Google Console to accept six of my sitemaps with SUCCESS.

The only one missing is https://breadtagsagas.com/attachment-sitemap.xml which comes up with an error message on the console and on sitemap validator. How do I correct this?

None of my posts or pages appear to be indexed but I am going through them slowly on URL Inspection and requesting indexing. Is this the correct approach?

Kind regards
Tony

Thanks for following up, Tony.

There is no attachment-sitemap in the sitemap index and visiting https://breadtagsagas.com/attachment-sitemap.xml redirects to your homepage. Did you submit an attachment sitemap separately? I suspect this could be a remnant from the old sitemap you had, https://breadtagsagas.com/sitemap.xml.

And yes! Using the URL Inspection tool and requesting indexing is the way to go. However, do note, that these steps may take a while and therefore requires some patience.

Do let us know how it goes.

• This reply was modified 3 years, 4 months ago by Maybellyne. Reason: one extra sentence

Thanks very much for all your help. For me it has been incredibly useful. I’ve gotten over the hump in re-indexing individual posts but there are still a lot to go. Google allows me 15 to 17 per day.

I don’t know how paranoid about Sitemaps I’ll be from now on!

I don’t if there is any way to recover the six years of Indexing? My 3 EH Carr kept being cited by thousands of university students (high on appropriate 1st pages). Now they can’t find the articles and won’t for a considerable period.

Anyway, I much appreciate your support and kindly keeping on replying. It would have been much more stressful without your help and answering my stupid questions.

Very kind regards
Tony

p.s. Google sent me an email Re: [1-0340000031656] How was your Google Support Experience? with a survey link. I haven’t got anything against Google. But how bloody useless was their email. I’m putting dozens of requests to them. How would I know what 1-0340000031656 was? I mentioned this in the survey!

They did mention 1-0340000031656 in 2 previous emails but as it was about a Googlebot being unable to access my site it was far from helpful.

Hi,

While sitemaps are definitely important (especially for bigger websites), they’re not the only way for Google to crawl and find new or updated content on your website. We recommend checking out this documentation regarding the index coverage report for you to know about the different statuses of possibly why your articles might have been excluded (either due to an error message or something else) and what you can do to address it – https://support.google.com/webmasters/answer/7440203/

Best of luck in further optimizing your website!