Crawler understanding

  • Resolved amauryt

    (@amauryt)


    3 years, 7 months ago

    Hello,

    I can’t seem to understand how the crawler work.
    The first time I crawled my site, all the urls have been crawled.

    After some changes and parameters change, I purged all the cache and asked for a new crawl, but of my 353 urls, only 34 have been crawled and 131 are in “missing cache” state.
    here a screenshot : https://ibb.co/MBM7bDC

    Why the crawler did not crawled all my url and only 34 ?
    What does that mean ?

    Thanks a lot,

    Best regards

Viewing 5 replies - 1 through 5 (of 5 total)

  • Hi,

    It’s fine, nothing to worry about. “Cache Miss” simply means the crawler has done its job by warming up those 131 URLs. “Cache Hit” means the crawler gets to a URL but its cache already exists, so in your case, those 34 URLs had already been cached when the crawler got to them.

    Best regards.

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    blue status does NOT mean the crawler did not crawl

    it indicates the page was NOT cached BEFORE, but it is cached NOW by crawler

    Best regards,

    Thread Starter amauryt

    (@amauryt)

    Hello @qtwrk,

    Thank you very much for your answer.

    A last question :
    I have a TTL of 1 week. All 353 urls were green after the crawl.
    12 hours later a second crawl is done, and I only have 46 urls left in green, and 84 in blue. What does it mean ? And why the total is not 353 ? (0 in grey and 0 in red)

    Thanks again for the help !

    BR

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    please try enable the debug log to all , then run crawler or wait for some times , and see what is inside of /wp-content/debug.purge.log

    when purge call happens during the crawling , it will mess up crawler and force it reset to beginning

    Best regards,

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    I’m going to mark this topic “Resolved”, due to lack of activity.

    If you still need help, please feel free to re-open it.

    When re-open it, please also change the topic status to “not solved”

    Best regards,

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Crawler understanding’ is closed to new replies.