load cache after purge on quiet site, no crawler

If I run a free web-based broken link checker does this load my pages into cache, e.g. https://www.drlinkcheck.com/

Try this application:

https://home.snafu.de/tilman/xenulink.html

It is an application to check broken links and works like a crawler, but you can also use a text file with pre-defined URLs. This application is an official recommendation from LiteSpeed.

thanks for suggesting xenulink.
I did a little web searching but still have some questions.
. When xenulink crawls a page does the page get loaded into cache?
. Can it overload my server so that my viewers are slowed?
. can I control this by limiting the number of crawlers operating at the same time?

. When xenulink crawls a page does the page get loaded into cache?

Yes, for cache plugin it doesn’t matter who or what does the request.

. Can it overload my server so that my viewers are slowed?

Yes, but to prevent a high load go to settings and set parallel threads to max. 3 threads. This setting is the same like in cache plugin for crawler, but the different is that cache crawler reduces the number of max. parallel threads if current load is higher than 1. In Xenu it is always fixed, because Xenu doesn’t know the current load.

. can I control this by limiting the number of crawlers operating at the same time?

Yes, see my previous answer.

Thanks very much for providing this xenulink option and answering my questions.

Running a broken link check of my 1200 links each week seems too much. Is there a similar program that just crawls and loads cache?

It seems to me that because of frequent plugin updates with the cache purge, any site with few visitors needs to have a crawler to get pages loading fast, with the resulting improved search engine ranking.
** Have I got this right?

I’ve just seen the plugin “warm cache”
What do you reckon?

Sorry, I don’t understand what your problem is?! With Xenu or build-in warmup crawler you get all what you need to provide cached URLs. Any other routine that does the same like Xenu or cache plugin crawler does the same, so you are turning in a circle. In your opinion what would make you happy?!

Hi,

so is there some way I can:
. define a group of my important pages and
. click a button to load these into cache after an update.

If you have something like cPanel , that allows cron job , you can try create a shell script to mimic curl call to your pages, then set the cron job for this script to runs over your site/pages every now and then to keep cache warm

Best regards,

I plan to use xenulink to load my cache. I will be very pleased to have regular checks of my 1200 links – although the check after each update will make this unnecessarily frequent.

I am interested in your suggestion qtwrk. I do have Cpanel, I have heard of “cron jobs” but I have never heard of a “shell script” or “curl calls”.
Can someone provide a suitable shell script to do this and an idea of how to use the script?
Presumably, this script would efficiently load into the cache a defined set of URLs.

Hi,

a quick script , not tested , not sure if it works or not , but just give you some hint and idea how this thing works

go to your file manager , create a file , like cache.sh with this code

https://www.litespeedshare.net/s/gnxxk5dm

replace part “www.your_domain.com” to your actual domain , and /home/username/url.txt , the part username to your actual username (you have total 3 places to change in this code , this won’t work if you miss any of them )

then create a cron job like once per day in your cPanel , with command (this is ONE LONG LINE , but forum breaks it show as in 2 lines )

bash /home/username/cache.sh https://www.your_domain.com/sitemap.xml

where the script will fetch the link list from your sitemap.

be aware , this will mimic as normal visitor to your site , by that means , the bandwidth still counts , and such quick , repeat visit may leads abnormal resources usage , check with your provider’s AUP first and use it at your own risk , I put a sleep 5 to make it run one page each 5 seconds , hope to ease some load.

Best regards,

• This reply was modified 3 years, 9 months ago by qtwrk.
• This reply was modified 3 years, 9 months ago by qtwrk.

You warn that this will mimic a normal visitor.
Does this mean that my google analytics statistics would be increased by one page-view for each of my pages each time I ran this? That’s a concern.

Thanks for your very full reply.

You warn that this will mimic a normal visitor.
Does this mean that my google analytics statistics would be increased by one page-view for each of my pages each time I ran this?

Yes, it mimics a user request, but Google Analytics doesn’t notice it.

I have one page on my site which links to all my other pages – a visitor site map. To load my pages into the cache, I only need to crawl this one page using Xenu. I want to limit the intensity of resource use on my shared server, so I could use:
. URL = https://www.mysite/visitor-map/
. Do not “check external links”
Under “more options”
. Parallel threads = 2
. “Maximum depth” to crawl this one page only = 0 (or it could be 1)

The curl program suggested above will be more efficient, but I do not know the name of the programming language, or what it is operating on, or how to test the program.
Out of curiosity, what is the name of the programming language?

https://en.wikipedia.org/wiki/Shell_script

https://linuxhint.com/30_bash_script_examples/

Now that is a whole universe of new knowledge. Thanks

enjoy it ??