slowing down

I’m using wordpress multi-user but it seems like the more I am on the site the slower my site gets. I recently increased the memory to 512 mb and only run 1 small site right now, while I work out the kinks. I’m not running many plugins or large images. I can’t figure out why it loads slowly. When I restart the server (rackspace) it speeds up but the problem starts again. I turned off some of the plugins but still run into the problem mentioned. Anyone have any suggestions? website is https://www.chatsworthhomesearch.com