I’ve noticed that the main delete button only clears the super cache files. I’ve been able to delete the regular cache one by one, by looking in the list view.
Although, sometimes some regular cache files will not delete this way either and I have to do it from FTP / SSH.
UPDATE: The manual delete not working, might have just been because there was a spammer hitting the same pages over and over. Since I deleted that account and blocked their IP I can delete all the regular cache files manually.