Hi Treetex,
Thank you for the proper explanation,
Yes, you can change the profile picture from your My-account/ Edit Profile. If the account has already set the gravatar, the remove button will display the default gravatar set by the user. If not, then it will display the blank. This feature is appropriately working on our testing and dev sites. We have manually tested and confirmed it.
If the remove button shows the same picture after reloading the site or changing the tabs, it can be cache or the plugin conflict. We suggest you clear the caches once. If this does not help, please deactivate all other plugins except User Registration and see if it works. If everything works fine, then you can activate other plugins one after another and do remove profile pictures at the same time. This will let you know the cause of the error.
Let me know if it helps or not, and I will get back to you.
Regards!