Looks like it’s your PHP environment.
Look at this Pingdom Tools output for your site.
See the first line? That’s the initial page generation. PHP took 4.2 seconds to generate that page and the PHP-speedy stuff is not helping you. Do you have a PHP op cache installed such as EAccelerator or Xcache? Those frequently help.
Disable all of your plugins (except the anti-spam ones such as Akismet) and switch to the default theme. Now with that default theme, re-run the Pingdom test again.
Did that improve anything? If so, put back your theme and repeat adding your plugins until you find out what’s causing that delay.