Prevent creating multiple sizes on image upload

I am using
add_filter('wp_handle_upload_prefilter', 'my_pre_upload', 2);

to change the upload directory when users upload profile images via my PlugIn on the WP profile pages (profile.php and user-edit.php). This is working, but since it is just a profile image, I only need the size the user uploads.

How can I prevent the creation of all the different sizes of just these images? I know how to change it for all uploads, but I only want to change it for the uploads coming from the profile pages in the backend (going to uploads/author).

Any ideas?