crawler miss everytime

  • Resolved olgamm

    (@olgamm)


    1 year, 1 month ago

    Hello,
    I have an issue with the crawler.

    Server setup:
    Web server Litespeed
    redis
    elastic search
    My $_SERVER[‘X-LSCACHE’] is on,crawler
    In my wp site, I have installed the Litespeed plugin, I enabled the basic settings, and enabled the crawler from the server and the plugin settings.
    I added the sitemap from the Yoast SEO plugin.

    when I visit a page for the first time is a miss.
    In the map page of the crawler the pages is always blue when i click update map is gray.
    What can I do in order to work?

    My dev site is https://iama.24lc.gr/

    thanks in advance,
    Olga

Viewing 15 replies - 1 through 15 (of 19 total)

  • The map is a snapshot showing the cache status after crawling. If the cache of a URL is changed by any actions, then this changed status will not be displayed in the map, which is technically not possible.

    Plugin Support qtwrk

    (@qtwrk)

    please provide report number

    you can get it in toolbox -> report -> click “send to LiteSpeed”

    Thread Starter olgamm

    (@olgamm)

    Hello,
    Report number TRVEKOSA.

    Can I enable ESI and crawler or it is an overkill?

    Plugin Support qtwrk

    (@qtwrk)

    emmm? what exactly is your server stack ? why am I even seeing varnish in it ?

    Thread Starter olgamm

    (@olgamm)

    Oh yeah, we had varnish before we installed the litespeed.
    I should disable it, do you think that is the problem?

    Thread Starter olgamm

    (@olgamm)

    Hello, I disabled the varnish.
    and I sent another report with the number: PAIJQHHX

    I ran the crawler manually but nothing changed.
    Thanks in advance.

    Plugin Support qtwrk

    (@qtwrk)

    is 95.xxx.xxx.116 your origin server IP ?

    and please lower your crawler interval from 302400 to something like one hour or two , otherwise crawler is not going to run again in 3.5 days

    Thread Starter olgamm

    (@olgamm)

    Yes, it is.
    I changed the crawler to two hours.

    Plugin Support qtwrk

    (@qtwrk)

    is the server IP works ?

    because I tried on several servers 

    [root@server1 ~]# curl -I -XGET https://iama.24lc.gr --resolve iama.24lc.gr:443:95.xxx.xxx.116 -k -v
* Added iama.24lc.gr:443:95.xxx.xxx.116 to DNS cache
* Rebuilt URL to: https://iama.24lc.gr/
* Hostname iama.24lc.gr was found in DNS cache
*   Trying 95.xxx.xxx.116...
* TCP_NODELAY set
* Connected to iama.24lc.gr (95.xxx.xxx.116) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

    it failed to establish HTTPS connection

    Thread Starter olgamm

    (@olgamm)

    This is the IP address I use to SSH to the server.
    Did you try to use http instead of https

    Plugin Support qtwrk

    (@qtwrk)

    no , your website is set to use https:// , then crawler will follow it

    Thread Starter olgamm

    (@olgamm)

    ok, i will try to check what i can do about that and i will come back

    Thread Starter olgamm

    (@olgamm)

    Hello, again. I think the curl command is not working because of the CDN. The site is proxied. Is there a way to make the crawler work with a proxied site?

    Thread Starter olgamm

    (@olgamm)

    I just tried the same command on another site with CDN and it’s working correctly. So maybe it is something with the litespeed configuration?

    Plugin Support qtwrk

    (@qtwrk)

    I can not tell from server side , but it looks like you are doing SSL offload or something ?

    try set drop domain name in crawler sitemap setting , and see if it helps.

Viewing 15 replies - 1 through 15 (of 19 total)
  • The topic ‘crawler miss everytime’ is closed to new replies.