I don’t really know of any other way to check if something else is changing the quality without running the debugger from the EWWW Image Optimizer plugin. You would have to check every setting of every plugin that does anything remotely image-related, and all the settings for your theme. And even then, it’s feasible one of them could change the quality without allowing configuration. That wouldn’t be very nice, but it’s possible.
“Is there a way to resize images without creating extra artifacts?” No, you can’t completely get rid of artifacts, that’s just a natural by-product of the down-sampling. You can possibly reduce them in a couple ways:
1. Increase the JPG quality level.
2. Make sure your server has the Imagick extension available (which is usually a little better than the GD extension used by default). You can check this under Tools->Site Health->Information->Media Handling
“Why is compression harder if we use Imsanity to resize?” — I’m not sure I understand what you mean there? Do you mean that you get worse results from your image optimization plugin after an image has been resized by Imsanity? If so, that’s pretty natural, since we’ve just chopped out a lot of the room for compression by scaling the image down AND if you’ve left the quality level at 82, the image will potentially be better compressed than the original you uploaded.
Overall, you’ll have achieved a lot more savings, but often the greater portion of the reduction is done by Imsanity when you scale images down.
Regarding the 5304px image, you said it was a PNG, and that is pretty much the reason, but I’ll try to explain why. The PNG image format is lossless, and stinks really bad for photographic content. It excels in images with a very minimal number of colors. When you scale an image down, you add more colors because neighboring pixels get merged/blended together in the process. More colors = worse compression for any format (this is where some artifacts come from), but it’s dramatically worse for the PNG format, to the extent that even a 1000px PNG might be larger than the 5304 version.
Let me know if I missed anything else!