Sitemap and urls issues preventing Google indexing

Can you please confirm that the plugin is connected to Google Search Console? We have more information on this here: https://kb.yoast.com/kb/how-to-connect-and-retrieve-crawl-issues/

In regards to the sitemaps, can you please let us know what the URLs that are being reported as inaccessible are so that we can check into them further?

thanks for your reply – it has revealed some new critical issues which I hope you will be able to resolve quickly.

Connection to my Search Console.
As per your article, I checked the Search console connection. It showed as NOT connected asking me to get the Google authrisation code.

1. First issue – why has the plugin connection with my search console dropped off? ? when did this happen?? It was definitely connected, I remember getting and successfully inserting the authorisation code several months ago. Did it drop off following one of the MANY plugin updates?

2. Ok, I requested the code again, copied and pasted into the plugin. It confirmed the successful connection showed however the WRONG property: http instead of https which is my property in the Search Console. Does it matter which prefix is listed, anyway? I ask, as per your article http and https are two different properties. My Property is https://www.quantumliving.com.au/ – by the way, what’s the deal with that final forward slash that most URLs seem to have these days? Is it important? I do know that if I redirect one I have to redirect the other too, otherwise it returns 404 error.

3. I requested re-authorisation. After few glitches and errors, I was finally able to save the correct profile.

4. Now my Seach Console tab is showing 26 x 404 crawls errors and 1 x soft 404 – which is what the Google Search Console (SC) shows. Since the posting of my first post I have cleaned up my SC in the desktop version – which I was primarily looking at. I removed from Google indexing all non-existing pages and created redirects for those that changed permalinks. Then requested re-indexing which showed zero errors for Desktop and still 27 Not Found for Smartphone – some of which by then had active redirection. I had also several Access Denied errors which I marked as fixed too quickly.

5. Yesterday I received an email from Google saying that from now on only smartphone crawls will be indexed and used for ranking of my website. Great. I have to fix these errors fast.

SITEMAP ERRORS

1. Google shows 1 error and 35 warnings across 8 sitemaps as it was unable to access the various pages from the sitemaps and could not access one sitemap at all. Some erros are strange as these pages can be accessed directly, which points to the sitemap issue. Index errors from sitemaps – 11 warnings (urls not accessible) and 1 error url (sitemap) not reachable.

2. I prefer not to list the urls for my sitemaps here on a public forum. Too risky. Could you please give me an email address or some other private form of communication where I could send you this detailed information?

3. I can tell you here that out of 159 submitted pages Google was able to index only 112. Out of 62 images it indexed only 5 – but that’s ok, I usually block indexing to all my images stored in the library so I don’t worry about it.

I will be able to fix the crawl errors for smartphone, but I do need your help to fix the sitemap errors please.

I look forward to reply.

hi @devnihil

I would much appreciate your reply with an email address or if could send me an email to info[at]quantumliving.com.au so that I could send you the sitemaps with errors and all the details.

I just ran a Ryte test which also showed unecessable sitemaps marking it as critical to my site indexing, which means that I need to fix it ASAP.

thanks

Hi,

You identified several issues. We have responded to them below.

A. First issue – why has the plugin connection with my search console dropped off?

We are unable to determine when the SEO-Search Console connection drops. We do know it is not expected to drop as the result of an update. We can only say to monitor the notifications in the SEO-Dashboard-General section to see if it happens again.

B. k, I requested the code again, copied and pasted into the plugin. It confirmed the successful connection showed however the WRONG property: http instead of https which is my property in the Search Console.

Yes, it does matter. You should add your site (including the www and the https) in your Google Search Console. Then re-connect it to that property using this guide: https://kb.yoast.com/kb/how-to-connect-and-retrieve-crawl-issues/.

Note, that if you have the http as a property in your Google Search Console you should keep it BUT delete any sitemaps you have under it. Then add your https and www site as a property and submit sitemaps to it. This guide explains more: https://kb.yoast.com/kb/submit-sitemap-search-engines/#google

Also, it does not matter SEO wise if you www or non-www. It is a personal preference.

C. by the way, what’s the deal with that final forward slash that most URLs seem to have these days? Is it important? I do know that if I redirect one I have to redirect the other too, otherwise it returns 404 error.

The trailing slash (that’s the term for it) at the end of URLs is personal preference. It is controlled by rules in your server. You can have it or not have it BUT the redirects should work correctly.

We see that the non-trailing slash urls get redirected to the trailing slash. So it is working correctly.

Google explains more here: https://webmasters.googleblog.com/2010/04/to-slash-or-not-to-slash.html.

D. Google shows 1 error and 35 warnings across 8 sitemaps as it was unable to access the various pages from the sitemaps and could not access one sitemap at all. Some erros are strange as these pages can be accessed directly, which points to the sitemap issue. Index errors from sitemaps – 11 warnings (urls not accessible) and 1 error url (sitemap) not reachable.

We can confirm your sitemaps appear as expected. Google is expected to crawl them. As we mentioned in a previous reply please delete and then re-add your sitemap and monitor it.

E. I can tell you here that out of 159 submitted pages Google was able to index only 112. Out of 62 images it indexed only 5 – but that’s ok, I usually block indexing to all my images stored in the library so I don’t worry about it.

Google may not index all of your content and urls. This guide explains more: https://support.google.com/webmasters/answer/34441?hl=en.

However, for this particular issue, we suggest re-adding the sitemap as described above and see if the issue resolves.

• This reply was modified 6 years, 2 months ago by Pcosta88.

hi @pcosta88

thank you for your reply.

I have reconnected my site with Google and now the property shows with the correct protocol (https://www)

Re the trailling slash – actually the redirects do NOT work for both. I use the Redirects plugin in which I have to include every redirect twice – with and without the trailling slash – as people put either form in the url field and as I found out for example /about/ will redirect propely however /about will return the 404 Not Found error – which is surprising. Any thoughts on that?

re the sitemaps – I have rebuilt and re-submitted my sitemap index to my Search Console, which removed the 32 warnings. However, there are still 2 errors remaining with 403 returns on two sitemap subcategories. These urls are listed as www. only – so perhaps that’s the problem. They show correctly in my sitemap list. Any idea what could be the issue? I have posted this question also on the Google Search console forum.

It’s amazing how much time webmastering takes, especially when you don’t have a webmaster and need to learn everything from scratch and do it yourself! ??

• Unfortunately, we’re unable to confirm why the redirects are not working when they don’t have a trailing slash. We recommend you please reach out the relevant plugin support for assistance.
• If the URLs are appearing fine in the sitemap but showing an error in the Google search console, we recommend you please resubmit the sitemap and wait for a while until Google recrawls the sitemap.