WP Multisite Backend issue

I have some weird things going on with my Multi-site since loading 4.1. For the last couple weeks I have been deducing the problem, but whats really interesting is the problems only happen to the main site – all child sites work perfectly on front end and back.

Here is what happens – Once I log into the main site back-end, I start getting connection lost errors. The front page takes forever, then errors out. Eventually the main site comes back up – especially when I am out of the back-end.

Keep in mind – if I go to a child WP site, it loads WITHOUT PROBLEM. I can even go into the backend and work without lost connections. All while my main site is being problematic.

I have removed all the plugins and refreshed the Plugin Cache. I have renamed the plugin folder. I have played with other themes. I ran a plugin diagnostic and kicked out any plugins that were taking up major resource.

I have also recreated my .htaccess since sometimes it shows up as 500 errors.

I am thinking about going back to 4.0 simply because of this issue. Does anyone have any other thoughts?