Adding Custom Sitemap by Yoast SEO

Hi,

Please make sure your cron is working properly

and you can also try start it manually for the first time to check if crawler is working fine or not.

Best regards,

Cron is working correctly.

When I do not use Custom Sitemap the crawler works fine and only crawls my blog posts. Does not crawl products and pages.

I’m using Woocommerce btw.

I’m not sure how to attach screenshots here but here is what is displayed from the Crawler side

Start watching…
1 June 2020 22:00:57 Size: 0 Crawler: #2 Position: 1 Threads: 5 Status: end

Sitemap URL domain.com/sitemap_index.xml

Hi,

what if you click “manual run” ? you may need to click it twice

Best regards,

When I click on Manually Run [R], I get the following Notice

Sitemap created successfully: 0 items

And the Crawler Log returns the following:

Current sitemap crawl started at: 12s ago

The next complete sitemap crawl will start at: 06/06/2020 08:42:31

Last complete run time for all crawlers: 5 seconds

Current crawler started at: 7s ago

Last interval: 7s ago

Ended reason: end

Any ideas what might be causing.

I tried disabling all plugins except Yoast, Woocommerce and LS Cache but still same issue.

Hi,

Could you please go to admin area , left-side bar —> LiteSpeed Cache —> Toolbox —> [6] Debug Settings

in this page:

set Debug Log to Admin IP Only

add your IP into Admin IPs (you will see your IP below.)

set Debug Level to Advanced

save changes.

Now go to [7] Log View, click on admin-bar LiteSpeed Cache —> Purge All

then click button [D] Clear Log , keep this window open, don’t close it nor navigate away from it.

Now ,run manual crawler for twice

then go back to Log View window and refresh this page, you should see the debug log is showing up.

Then please copy all the text , paste it into https://paste.ubuntu.com/

then post the link it generated for you.

Best regards,

Here is the link from Paste

https://paste.ubuntu.com/p/bgkTw8znFd/

Hi,

The log is flooded with CDN log

Please disable CDN feature for moment and do the debug log

Best regards,

CDN Disabled.

https://paste.ubuntu.com/p/2P7N3FGsB7/

Hi,

06/02/20 21:14:14.770 [94.46.150.33:52972 1 HOz] ?? ……crawler manually ran……
06/02/20 21:14:14.770 [94.46.150.33:52972 1 HOz] ?? ……crawler started……
06/02/20 21:14:14.773 [94.46.150.33:52972 1 HOz] ?? TouchedEnd. regenerate sitemap….

it seems crawler just finished itself immediately

Could you please try regenerate the crawler map and try manual run again ? with debug log

Best regards,

Hi, @storepaperoomates

I’m going to mark this topic “Resolved,” due to lack of activity.

If you still need help, please let us know what happened after you regenerated the crawler map.

We’ll be happy to reopen the topic if necessary.

Thanks!