Sitemap only has 2608 pages, website has 8000+

When I disable the “Output optimized sitemap” feature, instead I get one that is wp-sitemap.xml — which seems like the full complete sitemap. The problem is the URL is not the standard sitemap.xml so it might not be easily discoverable by search engines?

Howdy!

The Optimized sitemap reduces the number of items in the sitemap to make it more efficient. This helps search engines process the sitemap more quickly.

Search engines have a crawl budget for every website, meaning they stop crawling after several pages per day. If you include pages in the sitemap nobody wants or needs to read, such as your archives or old content, you might be pushing your good content to the back of the line.

The number of items in the sitemap corresponds to the “Sitemap Query Limit” value, which also affects the non-Optimized sitemap.

It’s important to note that for most websites, a sitemap is an outdated tool; search engines can find almost all pages via crawling. WordPress is well set up for crawling by automatically creating archives and meta-links. A breadcrumb is important to improve this further.

Search engines can discover the sitemap URL via the /robots.txt output, so the non-Optimized sitemap can also be found this way. In some cases, it’s might not be available there, and you’d have to submit your sitemap to Google and Bing manually.

You can inspect the pages Google has found on your site via Google Search Console’s Indexing Report.

I hope this explains most of the issue. You can learn more at https://tsf.fyi/kb/sitemap.

I use a caching plugin called NitroPack that uses my sitemap to find and cache all my pages. Is there a way to get the full sitemap generated for NitroPack — while still getting the smaller optimzied version sitemap.xml to use for search engines?

Hi again!

I’m afraid this isn’t possible.

Nitropack should not use the sitemap to warm up its caches; instead, it should rely on the WP Query API.

Reattributing something like the sitemap isn’t wise; if you try to appease independent, unrelated systems using a single tool, you’ll end up with a half-baked solution. It’s like using a screwdriver as a hammer; it doesn’t work the other way.

They probably did this because they’re not just a WordPress caching solution but also support other CMSes like Magento and OpenCart. All these CMSes use sitemaps, but only WordPress has the WordPress Query API. Not relying on the right tool for every CMS saves them development costs, but it comes with compromises.

I recommend using Nitropack’s standard caching warmup system. But if the sitemap-based warmup indeed proves beneficial, you can also consider using the non-Optimized sitemap.

Ultimately, the sitemap only helps speed up indexing, which is only vital to get right when you’re serving a lot of timely content, as news websites do. The sitemap is not particularly beneficial after every post or page is indexed, because they do not contribute to ranking.

• This reply was modified 2 months, 2 weeks ago by Sybre Waaijer. Reason: more info

Ok. I ended up using non-Optimized sitemap for now, and submitting it to Google and Bing too.

I found a strange issue. One of my sitemap files generated by SEO Framework loads perfectly fine in any browser (Chrome, FireFox, etc).

But it gives 404 error in Google Search Console, Bing, NitroPack, and every other 3rd party tool I use to try to access it.

It’s the one called wp-sitemap-taxonomies-product_cat-2.xml
All the others are working fine. Strange.

You can check it here:
www[.]swiftink[.]com/wp-sitemap-taxonomies-product_cat-2.xml