Manually changing uploads folder for specific categories

I understand that there isn’t really a way to dynamically change the uploads folder for media uploads to distinguish between categories.

Will I run into unexpected issues if I manually set the “uploads folder” prior to each batch of media upload? Will WP try to move previously uploaded media files? Will the “media library” section “forget” the files I’d previously uploaded?

Example: I’m working on the “apples” category of the website. Before I start uploading files that apply to that category, I change the media uploads folder to “wp-content/uploads/apples”. When I’m done working on that section, I change the media uploads folder to “wp-content/uploads/oranges”.

I realize it’s a kludge, trying to see if this could be a viable solution.