TMM Technology
Forum Replies Created
-
Glad to see you again @19ash82 ??
In the first place, I don’t think you will be billed for this, because:
a) The plugin visits the pages without JavaScript.
b) With the same IP (of your server).
c) It would be a severe loophole to make people bankrupt by simply running in the loop:curl -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" https://www.lesfousdisent.com/So I would recommend contacting the support of the service you are using to make sure making such a request will not cause you to be billed. I’m pretty much sure that you won’t be billed for this, and therefore you should not worry. But better to ask anyway, to be 100% sure ??
Anyway, I will also respond to all your questions one by one:
————-
> a) Does each crawl interpreted as one visitor or each page crawled as one visit ?
As said earlier, I doubt they bill for every visit blindly.
But it certainly depends on the service you’re using and their billing model. I would recommend to consult their support, with the code I sent above ??
> b) Since this would prove very expensive, the consent management platform proposed that your bot adds a Query string to the url and then I could condition the consent management to be activated in the absence of this string only.
Maybe you could consult their support if it’s possible to exclude by IP, and pass the IP of your web-server in there instead of the custom URL param?
Because passing a custom URL param can certainly bypass many caches (the cache won’t be created when the URL param is set) and therefore nullify the purpose of the plugin.
> Would the crawl still work & get recognized by Litespeed cache or other caching plugins?
It actually depends on the cache solution and its particular settings. I didn’t have experience with LiteSpeed.
But a rule of thumb: better not to pass any URL params whenever possible, as, from my exprience, many cache solutions do not create cache (or create it only for these URL params, and not for the page without them). This is why I proposed a way of exclusion by IP, as the most ideal.
> c) If yes, could you please add some option in the setting to add some query string please ?
There is such a setting in this plugin: Crawler Settings -> URL Params.
> Or should I add something in user agents in useing, for example “mybot” and then block my analytics if the word “bot” is detected in the user agent ?
Yes, that is another way. So I see you have these ways: query param, cookie, IP. But the first two ways can cause some caching solutions not to create the cache, so I would recommend to avoid them and focus on IP-based exclusion.@19ash82 Hi, we plan to make a documentation in the future, but for now I will be your documentation.
So the plugin goes from the entry points down the tree, like a very simple crawler. The depth is how many nodes to go deep (0 means visit only the entry point).
I understand that this plugin can be a bit hard to understand, we apparently will make some documentation later, but for now you can ask me if you need some help to make a specific setup.
Can you try to run with these 3 settings, as on my screenshot?
It works pretty succesfully for me:
@estalhun This feature is released. Thanks for your patience.
I have also added a migration for your constant, so in theory there should be a 0 value for you in here (you can delete the constant now).
@cptkoolbeenz This feature is released. Thanks for your patience.
@loyukfai This feature is released. Thanks for the patience.