Latest Update resulting in Sending Issues

Hi Philip – Sorry to hear that you’re having these problems.

To clarify, your 900 users ought to be receiving *digests*, correct? So the lines indicating the problem are:

[18-Nov-2021 17:00:43] Beginning digest batch of type sum for timestamp 2021-11-18 17:00:41.
[18-Nov-2021 17:00:53] Beginning digest batch of type dig for timestamp 2021-11-18 17:00:51.
[18-Nov-2021 17:00:53] Finished digest run of type dig for timestamp 2021-11-18 17:00:51. Digests were sent to a total of 0 users.

because 900 user should have gotten ‘dig’, but the log is showing that 0 users have received it, while in fact 116 have received it? Am I understanding this correctly?

The 116-instead-of-900 problem sounds quite similar to what you experienced in the past, namely that a timeout is preventing the next set of digests from being sent.

It’s not clear to me how 116 people could get the message, while the log says that *no one* got it. Here’s the logic: https://github.com/boonebgorges/buddypress-group-email-subscription/blob/48444c2056e0c4ace11b3dd7f5fbfefe389a9524/classes/class-bpges-async-request-send-queue.php#L159 If a user is added to the queue, then the $total_for_batch count will be incremented. Contrarily, if $total_for_batch is *not* incremented – as indicated by the “0” in your log – then it’s not possible that the user was added to the queue.

This, combined with your observation about “old” entries in the “View queued digest items” tool, suggest that maybe there’s some cache pollution somewhere. Are you running a persistent object cache – Memcached, Redis, etc? Try flushing that cache, or perhaps temporarily disabling it for a digest run, to see if the problem persists.

Beyond this, I guess I’m unsure what to suggest about the mismatch between 0 and 116. You might rule out some possibilities by looking directly at the wp_bpges_queued_items database table instead of the ‘View queued digest items’ tool in the Dashboard.

Dear @boonebgorges,

I ran a ton of further tests and I’m a bit at a loss. I reinstalled the plugin, I deleted the old entries from the database (1800 items were stuck), I cleared all my caching, disabled my caching plugin, and ran multiple test emails. The issue is not just affecting digests, but also immediate group notifications.

If I look at the log:

[24-Nov-2021 12:21:36] Beginning batch of immediate notifications for 9590.
[24-Nov-2021 12:42:37] Beginning batch of immediate notifications for 9591.
[24-Nov-2021 12:42:37] Finished sending immediate notifications for 9591. A total of 1 notifications were sent over all batches.
[24-Nov-2021 12:45:19] Beginning digest batch of type dig for timestamp 2021-11-24 12:45:14.
[24-Nov-2021 12:45:19] Finished digest run of type dig for timestamp 2021-11-24 12:45:14. Digests were sent to a total of 0 users.
[24-Nov-2021 12:45:20] Beginning digest batch of type sum for timestamp 2021-11-24 12:45:19.
[24-Nov-2021 12:57:56] Beginning batch of immediate notifications for 9592.
[24-Nov-2021 12:57:56] Finished sending immediate notifications for 9592. A total of 1 notifications were sent over all batches.
[24-Nov-2021 13:00:10] Beginning digest batch of type dig for timestamp 2021-11-24 13:00:08.
[24-Nov-2021 13:00:10] Finished digest run of type dig for timestamp 2021-11-24 13:00:08. Digests were sent to a total of 0 users.
[24-Nov-2021 13:00:11] Beginning digest batch of type sum for timestamp 2021-11-24 13:00:10.
[24-Nov-2021 13:07:10] Beginning batch of immediate notifications for 9593.
[24-Nov-2021 13:07:10] Finished sending immediate notifications for 9593. A total of 1 notifications were sent over all batches.
[24-Nov-2021 13:10:02] Beginning digest batch of type dig for timestamp 2021-11-24 13:10:01.
[24-Nov-2021 13:10:02] Finished digest run of type dig for timestamp 2021-11-24 13:10:01. Digests were sent to a total of 0 users.
[24-Nov-2021 13:10:03] Beginning digest batch of type sum for timestamp 2021-11-24 13:10:02.

I can see that for each of the emails sent, a number remain stuck in the queue, around 62 – which is more or less exactly the number of emails missing from the 116 or so that in theory would be sent.

Looking at the code, I think I’m not sure how to set $max_execution_time to 30 properly, because if I understand the logic correctly, it’s already set as such, no? How do I set the value correctly so I can experiment with lower times? Changing the sensible default in here to 15 doesn’t change a thing, but maybe that’s not the right place?

Kind regards, Philip

Dear @boonebgorges,

I figured it out ?? I found another threat herewhere the solution was to set the size of the batches. Now that I’ve done it, all works as it should! See the log:

24-Nov-2021 17:18:58] Beginning batch of immediate notifications for 9601.
[24-Nov-2021 17:18:58] Finished sending immediate notifications for 9601. A total of 1 notifications were sent over all batches.
[24-Nov-2021 17:20:19] Beginning digest batch of type dig for timestamp 2021-11-24 17:20:13.
[24-Nov-2021 17:20:19] Finished digest run of type dig for timestamp 2021-11-24 17:20:13. Digests were sent to a total of 0 users.
[24-Nov-2021 17:20:21] Beginning digest batch of type sum for timestamp 2021-11-24 17:20:20.
[24-Nov-2021 17:20:49] Sent sum digests to 51 users as part of this batch. Launching another batch....
[24-Nov-2021 17:20:50] Beginning digest batch of type sum for timestamp 2021-11-24 17:20:20.
[24-Nov-2021 17:21:19] Sent sum digests to 51 users as part of this batch. Launching another batch....
[24-Nov-2021 17:21:21] Beginning digest batch of type sum for timestamp 2021-11-24 17:20:20.
[24-Nov-2021 17:21:48] Sent sum digests to 51 users as part of this batch. Launching another batch....
[24-Nov-2021 17:21:49] Beginning digest batch of type sum for timestamp 2021-11-24 17:20:20.
[24-Nov-2021 17:22:02] Finished digest run of type sum for timestamp 2021-11-24 17:20:20. Digests were sent to a total of 175 users.

I just have to remember to determine the batch size after each update to the plugin, unless you have a more elegant solution for that issue?

Kind regards, Philip

Hi @strothi – Sorry for the lack of response over the holiday season.

Can you share the exact changes you made to make this work?

The plugin tries to determine the maximum batch size on the fly, by looking at memory and execution time limits. But your experience shows that this logic is imperfect, probably because of the differences in hosting environments (access to ini settings, etc).

As such, I might consider adding a filter in the plugin that would allow users to override the automated batch-size calculation, and instead to force a maximum batch size. This would be pretty simple for me to do, and would allow you to have a simple plugin that would enforce batch size. But before I go through the work, I want to understand exactly what you did to the plugin ??

Dear @boonebgorges,

I did exactly what you explained in your post in the referenced threat. I included the following code after this line ??

if ( $this->time_exceeded() || $this->memory_exceeded() || $total_for_batch > 50 ) {

Now, if this could be made into a setting so I doesn’t break each time the plugin is updated, that would be fantastic ??

Kind regards, Philip