Litespeed crawler is not working at all

  • Resolved tusharrao

    (@tusharrao)


    3 years, 8 months ago

    Hi,

    Report: TQZVUJHG

    The litespeed crawler is not working at all I am trying to crawl it i have tried troubleshooting many ways but it just doesn’t work. We’re using cyberpanel + Litespeed and we expected everything to work hand in hand.

    Because of this the visitors cannot see new refreshed page. However the page caches only when the user actually visits the page and this page once stored never refreshes.

Viewing 15 replies - 1 through 15 (of 15 total)
  • Plugin Support qtwrk

    (@qtwrk)

    Hi,

    please screenshot me your crawler status page

    Best regards,

    Thread Starter tusharrao

    (@tusharrao)

    Hi,

    https://prnt.sc/10yibls so far nothing is hitting cache.

    Regards

    Plugin Support qtwrk

    (@qtwrk)

    I don’t think it is running , now please go to [2] map , do you see any URL there ? and in [3] blacklist , do you see any URL there ?

    Thread Starter tusharrao

    (@tusharrao)

    No nothing there no URL in Map or Blacklist.

    Plugin Support qtwrk

    (@qtwrk)

    go to Map , click “refresh”

    Thread Starter tusharrao

    (@tusharrao)

    Refreshed but nothing also i can see notice “No valid sitemap parsed for crawler.” but the site map is present also i tried mentioning the link but doesnt work.

    Plugin Support qtwrk

    (@qtwrk)

    please follow this guide to grab the log when it tries to refresh the sitemap

    it should log more detail information why/what happened

    Thread Starter tusharrao

    (@tusharrao)

    Hi,

    Below is the log.

    03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] ?? ——GET HTTP/1.1 (HTTPS) /wp-admin/admin.php
    03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] Query String: page=litespeed-crawler&LSCWP_CTRL=crawler&LSCWP_NONCE=8a15bb5616&litespeed_type=refresh_map
    03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] HTTP_REFERER: https://xxxxx.com/wp-admin/admin.php?page=litespeed-crawler
    03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] Cookie _lscache_vary: admin_bar:1;logged-in:1;role:99;role_exclude_cache:1
    03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] LSCACHE_VARY_COOKIE: wp-postpass_0c4a259829633f08e9a633a47513ae93
    03/29/21 04:37:51.404 [149.202.98.186:18352 1 iGf] LSCACHE_VARY_VALUE: +webp
    03/29/21 04:37:51.814 [149.202.98.186:18352 1 iGf] [Router] LSCWP_CTRL: crawler
    03/29/21 04:37:51.814 [149.202.98.186:18352 1 iGf] [Router] LSCWP_CTRL verified: ‘crawler’
    03/29/21 04:37:51.815 [149.202.98.186:18352 1 iGf] ?? Init
    03/29/21 04:37:51.815 [149.202.98.186:18352 1 iGf] [Router] parsed type: refresh_map
    03/29/21 04:37:51.832 [149.202.98.186:18352 1 iGf] ????? failed to read sitemap: cURL error 6: Could not resolve host: xxxxx.com
    03/29/21 04:37:51.832 [149.202.98.186:18352 1 iGf] ????? ? failed to parse custom sitemap: Failed to remote read https://xxxxx.com/sitemap_index.xml
    03/29/21 04:37:51.832 [149.202.98.186:18352 1 iGf] ????? Truncate sitemap
    03/29/21 04:37:51.851 [149.202.98.186:18352 1 iGf] ????? Generate sitemap
    03/29/21 04:37:51.868 [149.202.98.186:18352 1 iGf] [Ctrl] not cacheable before ctrl finalize
    03/29/21 04:37:51.868 [149.202.98.186:18352 1 iGf] [Router] get_role: administrator
    03/29/21 04:37:51.869 [149.202.98.186:18352 1 iGf] ?? X-LiteSpeed-Cache-Control: no-cache
    03/29/21 04:37:51.869 [149.202.98.186:18352 1 iGf] [Optm] bypass: Not frontend HTML type
    03/29/21 04:37:51.869 [149.202.98.186:18352 1 iGf] End response
    ——————————————————————————–

    03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] ?? ——GET HTTP/1.1 (HTTPS) /wp-admin/admin.php
    03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] Query String: page=litespeed-crawler
    03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] HTTP_REFERER: https://xxxxx.com/wp-admin/admin.php?page=litespeed-crawler
    03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] Cookie _lscache_vary: admin_bar:1;logged-in:1;role:99;role_exclude_cache:1
    03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] LSCACHE_VARY_COOKIE: wp-postpass_0c4a259829633f08e9a633a47513ae93
    03/29/21 04:37:52.261 [149.202.98.186:19084 1 v0K] LSCACHE_VARY_VALUE: +webp
    03/29/21 04:37:52.648 [149.202.98.186:19084 1 v0K] [Ctrl] X Cache_control -> no Cache ( Admin page )
    03/29/21 04:37:52.744 [149.202.98.186:19084 1 v0K] ?? Init
    03/29/21 04:37:52.766 [149.202.98.186:19084 1 v0K] [Ctrl] not cacheable before ctrl finalize
    03/29/21 04:37:52.766 [149.202.98.186:19084 1 v0K] [Router] get_role: administrator
    03/29/21 04:37:52.766 [149.202.98.186:19084 1 v0K] ?? X-LiteSpeed-Cache-Control: no-cache
    03/29/21 04:37:52.767 [149.202.98.186:19084 1 v0K] [Optm] bypass: Not frontend HTML type
    03/29/21 04:37:52.767 [149.202.98.186:19084 1 v0K] End response
    ——————————————————————————–

    03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] ?? ——GET HTTP/1.1 (HTTPS) /wp-admin/admin.php
    03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] Query String: page=litespeed-toolbox
    03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] HTTP_REFERER: https://xxxxx.com/wp-admin/admin.php?page=litespeed-toolbox
    03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] Cookie _lscache_vary: admin_bar:1;logged-in:1;role:99;role_exclude_cache:1
    03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] LSCACHE_VARY_COOKIE: wp-postpass_0c4a259829633f08e9a633a47513ae93
    03/29/21 04:37:58.470 [149.202.98.186:24752 1 cML] LSCACHE_VARY_VALUE: +webp
    03/29/21 04:37:58.878 [149.202.98.186:24752 1 cML] [Ctrl] X Cache_control -> no Cache ( Admin page )

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    03/29/21 04:37:51.832 [149.202.98.186:18352 1 iGf] ????? failed to read sitemap: cURL error 6: Could not resolve host: xxxxx.com

    this was the issue

    https://sw_s_.com/sitemap_index.xml

    was your sitemap setting ?

    try curl this URL fro your server SSH first , see if it works there

    Thread Starter tusharrao

    (@tusharrao)

    This is from local but server i have not yet tried.

    <?xml version=”1.0″ encoding=”UTF-8″?><?xml-stylesheet type=”text/xsl” href=”//xxxxx.com/wp-content/plugins/wordpress-seo/css/main-sitemap.xsl”?>
    <sitemapindex xmlns=”https://www.sitemaps.org/schemas/sitemap/0.9″&gt;
    <sitemap>
    <loc>https://xxxxx.com/post-sitemap.xml</loc&gt;
    <lastmod>2017-06-16T08:04:22+00:00</lastmod>
    </sitemap>
    <sitemap>
    <loc>https://xxxxx.com/page-sitemap.xml</loc&gt;
    <lastmod>2021-03-28T09:44:28+00:00</lastmod>
    </sitemap>
    <sitemap>
    <loc>https://xxxxx.com/product-sitemap.xml</loc&gt;
    <lastmod>2021-03-16T23:14:18+00:00</lastmod>
    </sitemap>
    <sitemap>
    <loc>https://xxxxx.com/category-sitemap.xml</loc&gt;
    <lastmod>2017-06-16T08:04:22+00:00</lastmod>
    </sitemap>
    <sitemap>
    <loc>https://xxxxx.com/xxxxxxxxx_slider-sitemap.xml</loc&gt;
    <lastmod>2019-01-10T16:29:35+00:00</lastmod>
    </sitemap>
    <sitemap>
    <loc>https://xxxxx.com/product_cat-sitemap.xml</loc&gt;
    <lastmod>2021-03-16T23:14:18+00:00</lastmod>
    </sitemap>
    <sitemap>
    <loc>https://xxxxx.com/product_tag-sitemap.xml</loc&gt;
    <lastmod>2020-07-28T10:45:05+00:00</lastmod>
    </sitemap>
    <sitemap>
    <loc>https://xxxxx.com/pa_pt_phone_model-sitemap.xml</loc&gt;
    <lastmod>2020-06-06T13:26:16+00:00</lastmod>
    </sitemap>
    </sitemapindex>

    Plugin Support qtwrk

    (@qtwrk)

    yes, I know , it loads by external

    maybe your server has DNS resolver issue or something , that’s why I asked you to curl it on server first

    Hello – I have the exact same issue. XML sitemap curls fine on the server.

    Log indicates “…failed to parse custom sitemap: Failed to parse xml {URL}”

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    @zorbs please create your own topic for your issue with debug log as above process

    Best regards,

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    I’m going to mark this topic “Resolved”, due to lack of activity.

    If you still need help, please feel free to re-open it.

    When re-open it, please also change the topic status to “not solved”

    Best regards,

Viewing 15 replies - 1 through 15 (of 15 total)
  • The topic ‘Litespeed crawler is not working at all’ is closed to new replies.