Crawler sitemap

  • Resolved mcdeth

    (@mcdeth)


    8 months ago

    Hello. I have shop with 15k products. Half is out of stock. I understand it doesn’t make sense to crawl all these products. Moreover, there are totally 400 subcategories in funnel. Main category or “shop all” category, can have hundreds of subpages. It also most likely doesn’t make sense to crawl (they’re noindex for google).

    I’m thinking of making own sitemap with in stock products only and as for categories, include only first 3 pages. Does it make sense and wouldn’t crawler follow canonical link to next page anyway?

Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Support qtwrk

    (@qtwrk)

    crawler only visits the links it directly extracts from sitemap.

    Thread Starter mcdeth

    (@mcdeth)

    Ok, thanks. So is my assumption correct, to don’t waste time crawling 200th package and out of stock products?

    Plugin Support qtwrk

    (@qtwrk)

    the crawler won’t know what page that is , stock or no stock , it only goes to that page it grabs from sitemap

    so if you have some “high priority” page that wants to be crawled , you should manipulate your sitemap list for it

    Thread Starter mcdeth

    (@mcdeth)

    Thanks!

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Crawler sitemap’ is closed to new replies.