wp always slow

My wordpress page is loading very slow – typical story. Yet it is slow even with fresh new wp instalation, with no plugins installed, no content, default twenty eleven theme and no traffic on the website. Time to first byte often exceeds 15 seconds, then files are then downloading very slowly – 5KB/s or slower. I’ve been strugling with this for a month, couldn’t find any solution. I checked the server, mysql is running fast – about 22 000 insert queries in a second on innodb tables, php is running good as well and when I install Nucleus blog software, it’s lightning fast! I still want to try make wordpress going, but if it is slow like this, I will have to go to joomla, or something.

Another interesting fact about it is that some people told me that it is going fast for them and some “check my website speed” internet tools say the same thing, but others say it’s slow, I see it being very slow.
Seems like internet or routing troubles from this point of view, yet theese problems occur only with wordpress. It seems very strange to me.

website name is theveryfirst.com, other website where i am curently testing the stuff is millionairegirls.com – both websites have the same problem.