Set Crawler for custom user agent

  • Resolved David

    (@miladk)


    1 year, 9 months ago

    Hello,

    i have added an user agent to the htaccess file (Yandexbot) as described here: https://docs.litespeedtech.com/lscache/devguide/usage/ :

    RewriteCond %{HTTP_USER_AGENT} Yandex [NC] RewriteRule .* – [E=Cache-Control:vary=isyandex]

    So far everything works fine and litespeed in generating seperate cache files for the Yandex bot on Dekstop and Mobile.

    My question is how can I add the Yandex bot to the crawler (Desktop and Mobile separately) since it isn’t available in the default variables and therefore doesn’t get crawled at all:

Viewing 1 replies (of 1 total)

  • My question is how can I add the Yandex bot to the crawler (Desktop and Mobile separately) since it isn’t available in the default variables and therefore doesn’t get crawled at all:

    The built-in crawler does not provide this option and is therefore not possible. The Yandex bot is still taken into account, or the cache is warmed up for it. Defining a cache vary extra for Yandex is therefore not necessary.

Viewing 1 replies (of 1 total)
  • The topic ‘Set Crawler for custom user agent’ is closed to new replies.