Unique visitors per crawl

Or should I add something in user agents in useing, for example “mybot” and then block my analytics if the word “bot” is detected in the user agent ?

But then would the cache still work or would the crawl be detected as a crawler ?

• This reply was modified 1 year, 2 months ago by 19ash82.
• This reply was modified 1 year, 2 months ago by 19ash82.

Glad to see you again @19ash82 ??

In the first place, I don’t think you will be billed for this, because:

a) The plugin visits the pages without JavaScript.
b) With the same IP (of your server).
c) It would be a severe loophole to make people bankrupt by simply running in the loop:

curl -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36" https://www.lesfousdisent.com/

So I would recommend contacting the support of the service you are using to make sure making such a request will not cause you to be billed. I’m pretty much sure that you won’t be billed for this, and therefore you should not worry. But better to ask anyway, to be 100% sure ??

Anyway, I will also respond to all your questions one by one:

————-

> a) Does each crawl interpreted as one visitor or each page crawled as one visit ?

As said earlier, I doubt they bill for every visit blindly.

But it certainly depends on the service you’re using and their billing model. I would recommend to consult their support, with the code I sent above ??

> b) Since this would prove very expensive, the consent management platform proposed that your bot adds a Query string to the url and then I could condition the consent management to be activated in the absence of this string only.

Maybe you could consult their support if it’s possible to exclude by IP, and pass the IP of your web-server in there instead of the custom URL param?

Because passing a custom URL param can certainly bypass many caches (the cache won’t be created when the URL param is set) and therefore nullify the purpose of the plugin.

> Would the crawl still work & get recognized by Litespeed cache or other caching plugins?

It actually depends on the cache solution and its particular settings. I didn’t have experience with LiteSpeed.

But a rule of thumb: better not to pass any URL params whenever possible, as, from my exprience, many cache solutions do not create cache (or create it only for these URL params, and not for the page without them). This is why I proposed a way of exclusion by IP, as the most ideal.

> c) If yes, could you please add some option in the setting to add some query string please ?

There is such a setting in this plugin: Crawler Settings -> URL Params.

> Or should I add something in user agents in useing, for example “mybot” and then block my analytics if the word “bot” is detected in the user agent ?

Yes, that is another way. So I see you have these ways: query param, cookie, IP. But the first two ways can cause some caching solutions not to create the cache, so I would recommend to avoid them and focus on IP-based exclusion.

Got to say, I’m loving your crawler & all the options in your plugin.

Yes, I can do exclusion by IP. I’ll do it this way. Thanks !

The consent management service actually emailed me I had exceeded my monthly quota in 4 days ?? I’ll have to go and find out how that was possible. Maybe other bots. I’ll probaby also add an exclusion if there is the word bot in the user agent.

@19ash82 I’m glad to hear you do enjoy it ??

If you have a moment, we’d greatly appreciate your?review. It not only helps others discover the plugin but also serves as a wonderful motivator for the development team.