wp_generate_attachment_metadata memory issue (Advanced)

I have a script that imports lots of large images from an external source into WordPress. I’ve run in to a memory issue that I am trying to debug. When triggered manually, the image upload works but the script dies at this point

$attach_data = wp_generate_attachment_metadata( $attach_id, $filename );
wp_update_attachment_metadata( $attach_id,  $attach_data );

So it seems that WordPress dies when trying to generate thumbnails. I am not able to upload the file manually via Media Library either. There I get

Fatal error: Allowed memory size exhausted in …/wp-includes/media.php

I was thinking it’s php memory limit causing it but the script does not fail when triggered by wp_cron.

I have tried setting WP_MEMORY_LIMIT and WP_MAX_MEMORY_LIMIT but neither seems to have an effect.

Any ideas?
Thanks in advance!