In my experience you need at least 1.5GB of RAM, and possibly 2GB, to run even a medium traffic WordPress web site with more than a few plugins.

You specifically mention Admin panel slowness. Admin panels take about twice the RAM of public parts of a WordPress web site.

And, yes, all of my VPS experience has been Ubuntu.