EWWW Bloat in WP_options

p.s. I have not removed these entries yet, so have at me for debugging ??

Ok, so a bit more background here, each entry is a “queue”, which may contain one or more image IDs. What we want to do is see what EWWW is doing about the image IDs in question, of which there will be a lot on your site. So, with EWWW’s debugging option enabled, please post the Debug Information from the settings page via pastebin.com.

Then, there is a debug.log file within the ewww-image-optimizer folder, but we want to wait a couple hours after debugging is enabled before we look at it, to make sure we get enough information. If it hits 1-2MB prior to that, we should have plenty of information, and no need to wait any longer.

We would also want to see if we can visit the Image Queue Debugging page, but I’m not sure it will load with 21,759 queues, so we’ll come back to that after we can check the debug.log file to see what it is doing.

The only time I’ve seen that magnitude of images get stuck in the queue (so far) was on a multisite install, so please let me know whether this is a multisite or regular WP install.

Lastly, the background processing uses async requests and wp-cron to do it’s job. If you have wp-cron disabled for some reason, then we need to go down a different road. As far as the async requests, I’ll be able to see if those are working (or at least if the test request worked) from the debug information I mentioned above.

Forgot to mention that you can send the debug.log file via https://ewww.io/contact-us/ if you like.

Great, thanks. Just got back into the office so I’ve enabled debugging and will send you the debug.log in a couple of hours. Thanks

I have now sent you the latest debug.log file using the site.

Thanks,
Thom

Ok, I’ll take a look here in a bit.

Alright, the debug.log looks very similar to the last time I saw this. It’s got a whole bunch of stuff stuck in the queue without any metadata, and it takes a while to flush those out.
If you can’t get to the queue debugging page, go ahead and remove all those ‘batch’ options from the DB, delete the debug.log file, and see if an image upload will process normally (within a minute or two after upload).

If it works, then I’m wondering if you have any sort of batch/automatic import process that pulls images in from an external source? The other site where I saw similar behavior had a batch importer that imported articles from several external sources, so I’m wondering if you have something along those lines that is clogging up the queue?

I just wanted to come back and say a huge thanks to you.

You are right, I couldn’t access the queue debugging page, even when bumping up the available memory and execution time (nightmare!), so I have removed the batch rows directly from the DB and the whole site is back to running smooth as butter.

Also, well done for spotting that we have an external feed, I understand how this plays havoc with the plugin. For the moment, we have disabled the plugin so as not to end up in the same situation until we have done some testing.

You, sir, are a superstar! Thanks for your great support.

Thom

What plugin are you using for the external feed? I’d be interested to see if I can figure out where the glitch is. The other site I had seen, it seemed like media would sometimes get inserted with one ID, and then re-inserted with another ID instead, which made the queue sit there and retry non-existent images until it timed out (15 retries).

The other thing I wondered, is if the workflow looks something like this:
1. the batch import runs, creating a whole bunch of images, and queuing them for optimization.
2. an editor comes along and deletes a whole bunch that they don’t want to keep

Rinse, repeat, and if that happens often enough, it could cause the backlog we saw. Or perhaps the “feed import” plugin will import 100, and then auto-purge several based on some predefined criteria?

Just throwing ideas out there to see what sticks, as I would love to be able to crack this one open.