Thanks for the suggestion Kevin!
I’ve tried modifying the htaccess that way as well as removing it entirely but unfortunately it doesn’t seem to help.
I’ve noticed that if I click on the google result and do a reload on the page I’ll still end up at the 404 site. If however I simply click into my browser’s location bar and hit enter, it will find the page I’m looking for.