Hi,
We have a to-do plan for something like online crawler , but right now it’s still at early stage and more like brain-child
if your provider allows cron job , you can create a simple shell script with code like this
#!/bin/bash
if [[ ! -f sitemap-urls.sh ]] ; then
wget https://raw.githubusercontent.com/yuriyyakym/sitemap-urls/master/sitemap-urls.sh
chmod +x sitemap-urls.sh
fi
rm -f urls.txt
./sitemap-urls.sh https://your_domain.com/sitemap.xml > urls.txt
n=1
while IFS='' read -r line || [[ -n "$line" ]]; do
curl $line \
-H 'authority: your_domain.com' \
-H 'pragma: no-cache' \
-H 'cache-control: no-cache' \
-H 'upgrade-insecure-requests: 1' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36' \
-H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'sec-fetch-site: none' \
-H 'sec-fetch-mode: navigate' \
-H 'sec-fetch-user: ?1' \
-H 'sec-fetch-dest: document' \
-H 'accept-language: en;q=0.8,es-ES;q=0.7,es;q=0.6,en-US;q=0.5' \
--compressed
let n++
done < urls.txt
change your_domain.com
to your actual one , there are 2 of them in code , make sure you changed both.
this script will mimic desktop Chrome to access the page , and generate cache for it.
Best regards,