I managed to keep the plugins, themes and content. Just deleted WP-Admin and uploaded the fresh version and the same for the functions.php. The rest was left as is.
For me this happened on BlueHost and I have a feeling it has to do with their auto script even though I manually installed this particular site.