WP is Limited to 10K Users

I have a WordPress site on a dedicated machine, SSD drives, 8GB of RAM and excellent processing power. Have 5 multisites. Once I reached 5,000 users the member admin crawls something awful. Users settings in admin set to only show 20 users at a time so thats perfectly cool. But clicking all users is like 10 full seconds to just get the first screen.

Decided to import a bunch of other users to see what goes. At 10,000 users the wait just to get to the first screen was a joke, like around 25 seconds. The posts and comment still work pretty good if youre surfing the site.

Plugins – almost nothing. Multisite, gravity forms (1 form) orphanage. Turning them off doesn’t make any difference. The users table just wont load. Database is optimized so it isnt that.

I think at about 5,000 users you just cant administrate any more. Really bad. Is there something that can be done?