Bulk Optimize Fails – Use Cloud API instead?

11,000 images is no big deal, I routinely run the scanner on 40k on my dev box ??

Try increasing the memory_limit and see if that helps with the scan issues.

Using the API does indeed offload the compression to the “cloud”, which is really a cluster of servers dedicated to compressing your images.

You can use either plugin, but if you decide to use the API, I recommend using the Cloud version, because it is much smaller, so it updates much quicker. Not that it’s too big of a deal, but I use it on most of my own sites now for that reason.

After you optimize everything, you’ll definitely want to purge your CDN cache to make sure they grab fresh copies of everything.

If the memory_limit thing doesn’t work, hit me up here: https://ewww.io/contact-us/

Thanks for the quick response. I’m on DreamPress (the DreamHost WordPress service). They have a dynamic memory allocation I believe and while I can ask to up the memory allocation, they might not do it.

So, if I were to switch over to the cloud version of the plugin, should I fully delete the server version (what I have now) or just disable?

And, moving forward, once I bulk optimize using the Cloud version and get everything optized (and purged from my CDN), would I then move back to the server version (which seems to work fine for small, article-related uploads)?

Thanks!

You might as well delete the server version if you switch to the cloud version. As for going back to the server version later, of course I’d rather you didn’t ??
But the main thing is what level of compression you use. If you’re just using lossless, then it is probably fine to use the server version for small uploads here and there. But if you take advantage of the huge savings that come with Max Lossy mode, there’s really no going back, it’s pretty fantastic stuff.
You can run a few test images at tinyjpg.com to try it out and see for yourself.

As for the scan issue, I’m going through some debug information from someone else, but if I strike out there, I’ll send you the instructions for further debugging on that. It really should NOT be a memory issue anymore, but I figured if you had the ability to try that real quick, it would be worth checking out, just in case.

Hi, I work for DreamHost (and I’m on the DreamPress team specifically).

What memory limits do you recommend for this? PHP limits can be adjusted within reason, of course, but I know even our shared starts at 90M and DP is higher.

The max execution time is also adjustable (within reason ?? ) but in general we don’t like raising that forever since it means EVERY process can run for a really long time, and that gets messy with some plugins.

Hey Mika, it depends on how large the library is, but as I told @hightechdad, I’m not sure it really is a memory issue. I built in extensive memory checks in the last couple releases, to try and prevent EWWW from exceeding the memory limit (with buffers of 2-4MB to be safe). On my local box I was able to scan a site with 40k attachments using only 16MB of memory.

As for time limits, the scan should almost never bail because of that, as it has a cap of 15 seconds for each batch of images. Every 100 image attachments, it checks both the memory usage and the time elapsed. So it will short-circuit the scan when it detects a resource constraint, and then resumes the scan on a subsequent AJAX call.

So, on the memory limit, if you could increase his settings by about 50%, I think that would allow us to determine for sure if it is a memory issue, or something else.

Also, if you know any other tricks for getting error output, I’d appreciate it. There seems to be some odd glitch we’re hitting, and I’ve had folks enable WP_DEBUG, and check error logs to no avail. I’ve resorted to adding extra logging calls within the scan functions to try and pin it down, but haven’t quite nailed it yet (and hopefully we aren’t still dealing with multiple issues causing the same 500 server error).

Awesome to have Mika on this thread as well. I just switched over from Server to Cloud version (purchased APIs).

First, I didn’t delete the Server version, I just disabled it. When activating the Cloud version, much of my same settings were active (should I have deleted Server version?).
Second, I believe DreamHost/Press updated my max_execution_time. I will let @ipstenu address the memory allocation.
Third, just tried running the bulk optimize. It’s still hanging (“did not complete, will try again”).

So, should I fully delete both plugins and start again from scratch?
Should I try a “force reoptimize”?

Looking forward to getting this working!

The two plugins share settings, that shouldn’t really be a problem, and it certainly doesn’t affect the scan either way. I’ve been helping 4 different people with the same problem, so I apologize if I repeat things, but can you let the scan run until it stops trying (or succeeds)?

Then once Mika bumps the memory, you can try it again just to see what happens. Some failures are not entirely unexpected with a library that size. It can happen if an image has corrupted/missing metadata, and EWWW will try to rebuild it, which has the side effect of regenerating thumbs, and potentially triggering whatever broke the metadata in the first place. When that happens, EWWW will throw an error, but it will reattempt the scan, and mark that attachment as ‘bad’ for future scan attempts.

If that (or something else) kills the scan 10 times in a row, then it will stop trying, but you’ll get a different error message when that happens.

Thanks. I let it run and got “Operation timed out, you may need to increase the max_execution_time for PHP” (which was already done). It’s set at 180 secs.

My memory_limit is 512MB currently

So, I can perhaps open a ticket with DreamHost (or maybe @ipstenu can just do it for me – grin).

If 512 isn’t enough, then … yikes. @hightechdad, what’s your domain? I can go look and see if it’s actually something else. Like our process watcher may be drunk again…

Yeah, 512 should be plenty. I can run the scan full-tilt locally with 256M and 40k attachments, so it’s got to be something else.

Thanks @ipstenu – yeah, well it could be the 20 million plugins I run. The site is https://www.hightechdad.com. Site is old (but current) and has mushroomed over time. 60-70k visitors/month. Lots of ads, plugins, images, you name it. It’s a bit of a beast so I have been trying to do some optimization – CDN, CloudFlare, minification, optimization, etc. Any help/advice is appreciated!

Ok, I have another thought for @ipstenu. I was looking at my error logs and there are quite a few instances of paired errors (different IPs) related to ModSecurity and eWWW.

Here are a couple:

`Feb 08 12:38:24 2017] [error] [client OneIP] Premature end of script headers: admin-ajax.php, referer: https://www.hightechdad.com/wp-admin/upload.php?page=ewww-image-optimizer-bulk
[Wed Feb 08 12:42:20 2017] [error] [client TwoIP] ModSecurity: Access denied with code 418 (phase 2). Matched phrase “WQGP” at ARGS:rmfilter. [file “/dh/apache2/template/etc/mod_sec2/99_dreamhost_rules.conf”] [line “151”] [id “1990070”] [msg “Common known arguments for backdoor shell present in ARGS:rmfilter”] [hostname “www.hightechdad.com”] [uri “/wp-load.php”] [unique_id “WJuCrNBhke4AABs3GDAAAAAF”]`

So, I’m wondering if the DreamPress firewall is viewing the upload.php call to the eWWW cloud as something as a security issue and is preventing the execution. I could be way off base here but I thought I would float it out there.

It’s only 86 plugins ??

So here’s your error!

Premature end of script headers: admin-ajax.php, referer: https://www.hightechdad.com/wp-admin/upload.php?page=ewww-image-optimizer-bulk

Technically yes, it’s PHP running out of spoons to run the process. But in this case it’s not memory. I mean, it’s not the 512M. I think you’re right that it’s the 86 plugins.

Edited to add: The reason you see modsec at the same time is that the failure it causing a weird thing to pass to modsec so it ‘looks’ like a hack

Looking at them, I would disable (for the purposes of this process only!)

* all-in-one-wp-security-and-firewall
* blogvault-real-time-backup
* regenerate-thumbnails
* wp-dbmanager
* wp-serverinfo

And you probably want to remove these:

* x-custom-404 (this is active!)
* x-disqus-comments
* x-email-mailchimp (this is active!)
* x-smooth-scroll
* x-typekit

Unless those are all special plugins for your X theme.

I saw you told WP to use 512M but not PHP, so I changed that for you and bounced Apache to clear up some hung processes. That MAY be enough? I hope… Otherwise this may be a ‘Turn off ALL the plugins’ because while I’m not an advocate of the ‘ZOMG! too many plugins!’ I’m familiar with some of those plugins and they can be a little ‘heavy’ to run.

• This reply was modified 8 years ago by Ipstenu (Mika Epstein).

That could indeed be it right there. I’m not quite sure I understand what it is saying though. @ipstenu, if you’d prefer to explain it via private email, I believe you have my address, otherwise https://ewww.io/contact-us/ so we can figure out how to avoid tripping that rule inadvertently.

We posted at the same time ??

So you can test by disabling it in panel: https://panel.dreamhost.com/index.cgi?tree=domain.manage&

Click on the edit and turn off ‘Extra Web Protection.’

But I don’t think this is the case seeing as we’re getting “Premature end of script headers” at the same time. If it was Mod Security, you’d get a 418 error.