I understand where your comming from. I have ran a test with a small html page on my host, and it comes up extremely fast, which i believe is normal webpages, but the WP section seems to be slow.
Less than 10 plugins running, I would imagine that is not a lot, although 1 could be an issue if done wrong.
Dunno too much about the rest tho. lol =) Blah, this sucks.