Cron stops working when feed runs

Hi svaldesm,

Thanks for reaching out.

45.000 products is a lot, especially if those 45.000 also contain variable products which each have variants as well. The error “error sending QUERY packet” is not something we have been seeing before with any other user so we did some research online and they do indicate a resource issue (memory, timeouts, etc) is going on. The strange thing is that you mention it has been working fine for at least a couple of days so what changed the last couple of days that could cause this all of a sudden? Did you install other plugins that are heavy on resources or did you make changes to the feed configuration?

Best,
Eva

Hi Eva, thanks for your quick answer.

How much products can this plugin support?
We don’t have variations, so it’s only the products (and it is suppose to filter them by “in stock”, in which case it will only show 26,0000).

We haven’t changed anything beside that. Do you have more customers using your plugin for so many products? Is this made for scalability?

Regarding the error, we are debugging further, but apparently it’s related to the plugin.

Thanks

We run the cron manually and there’s nothing wrong with it.
Is the workflow the same if it is triggered manually rather than automatically from the cron?

I see that the cron itself hasn’t been scheduled for updating again. I’ve turned on/off the plugin, on/off the feed, saved settings and nothings make it schedule.

Another thing weird is that, on the plugin Settings page, it says the WP-Cron is disabled, but I know it’s enabled on WP-Config and other cron schedules are working.

Any known issue with this as well?

Thanks

Santiago

• This reply was modified 5 years, 6 months ago by svaldesm.

Hi svaldesm,

The plugin is designed so it can handle large amount of products. That is why, for example, it processed the products in small batches. Every time a batch ended processing a new cron event is created to kick off the next batch 2 seconds later. This all works fine when the WP-cron is enabled but causes problems when you are running the WP-cron from your unix cron as the cron events sit idle until your unix cron is executed again.

For arguments sake lets assume your unix cron is executed every 5 minutes and you are running a PHP version of 7 or higher:

45.000 / 750 (batch size in your case) = 60 batches needed. In the most ideal situation it would therefor cost 300 minutes already just to execute all batches but since the products also need to be put in a feed this could easily become 450-600 minutes before the feed is done processing. That is why, even though it does work, advise against disabling the WP-cron in the wp-config.

Some of our users are well over the 100.000 products and have the WP-cron enabled. It works for them. However, this only works when having serious server hardware.

Best,
Eva

Hi,

Assuming the issue at hand has been resolved and our support is no longer needed I am going to close this topic for housekeeping reasons now.

Please don’t hesitate to reach out to us again whenever you need our help or have a question.

Have a wonderful weekend,
Eva

Hi Eva,

Thanks for your answer. There are two topics here that I would like to discuss:

1. Using server cron instead of WP-Cron: we’ve been advised many times to use server cron for several benefits regarding server performance. If adjusting server time to, say, 1 or 2 minutes, will this still work? Could you please elaborate more on how the plugin works in creating batches?

2. Plugin breaking some crons functionality and doesn’t update: when we manually update the plugin, it works. But the scheduled updates don’t work (Redis is not working on this server).
In fact, what’s even worse is that somehow other cron schedules suffer because of this, probably because something happens with this task that gets hanged.
For example woocommerce_cancel_unpaid_orders only gets scheduled after the last execution ends. It does run but at the same time that the feed should be updated, something happens and it doesn’t work more.
We have turned the feed plugin off and it does seem to work.
This is a 16GB server with regular traffic, it works very fast, so it’s not hardware related. Maybe a config issue might be breaking it, but there’s not much on the logs right now. Any clue on what can this be about?

Thanks a lot!

Hi svaldesm,

The plugin processes products in batches because when it would process all products at once and tries to write them to a feed all at once most servers would run out of memory or into timeout issue.

The default product batch sizes are as following:

• PHP version < 5.6 = batch size of 50 products at once;
• PHP version 5.6 = batch size of 200 products at once;
• PHP version > 7 and the number of products on your website is smaller then 50000 = 750 products at once
• PHP version > 7 and the number of products on your website is over 50000 = 2500 products at once

When you are done creating a feed or manually refreshed a feed it process the first batch. When one batch is not enough to process all products the plugin sets a cron event after finishing a batch to fire off the next batch of products. And so on and so on until it processed all products.

This cron events is set to fire a few seconds after a previous batch finished. Now when you trigger the cron events with an external cron the cron event sits idle until it is triggered. So whereas with the WP-cron enable there is only a few seconds between each batch in your case it depends on the interval you set for the external cron.

Hope this makes sense?

Best,
Eva

Hi Eva,

It does make sense and it’s clear to me for point 1.

What I’m not very clear is on point 2. As I wrote above, when the plugin runs “automatically” it somehow breaks other cron tasks. Could it be that RAM usage or something with server configuration makes it crash? Because the feed does not get updated. Please mind this is a very big machine and it handles lots of traffic perfectly, so it’s not a hardware issue (but maybe a config one).

How can we debug this further? Is there any known config (php memory, RAM, etc.) that makes the plugin “crash” when automatically updating? It somehow conflicts with something else apparently.
WP-Cron is enabled and Redis as well on this server and the feed is refreshed when manually done.

Thanks for your help,

Hi svaldesm,

With regards to point two: we haven’t had any other users informing us that our plugin breaks other cron tasks so I think you are right in assuming it conflicts with something else, possibly a plugin like Redis.

We have spend a lot of time in the past trying to make our plugin compatible with the Redis object caching but it caused so much unexpected behaviour that we decided to give up on it. Have you tried running our plugin with Redis disabled and if so did the problem persist?

Best
Eva

Hi Eva,

I’m sorry, I wrote that wrong. Redis is disabled on this site.

So I’m not sure how to debug the behavior of the plugin and why it breaks.
A yellow flag (maybe) is that according to the config of the plugin, WP-Cron is disabled. Not sure why that happens, because we can even see Cron Events with WP Crontrol (otherwise impossible if the plugin would be disabled).

If you know ways to debug or what to look for in the logs, that would be very helpful (are there any logs in the plugin?)

Thanks a lot,

Hi svaldesm,

Could you try disabling and re-enabling our plugin? It should reset the cron event that triggers the automatic updates. That should get rid of the message the WP-cron is disabled.

When our plugin would create errors or warnings those can be found in the WordPress debug.log (in the wp-content directory).

Best
Eva

Hi Eva,

Indeed it rescheduled the crons. I’ll wait a few days with the feed without being updated to see if it’s the feed or something else that breaks the cancel_unpaid_orders cron.
In case nothing else breaks it, I’ll turn on again automatic updates of the feed and check if that is the cause of the issue.

Thanks a lot for your help and I’ll keep you posted to further diagnosis.

Hi Eva,

After turning the plugin off/on, the crons did activate again.
But when the feed was suppose to update, it never did and now the crons for the plugin are gone.
Any idea on how to debug this?
WP-Cron is active, Redis is disabled, we have many visitors so Cron runs all the time.

Thanks,

Hi svaldesm,

Could you reach out to [email protected]? We will take it from there.

Best,
Eva