One thing that might cause “slowness” on some requests is if your server is behind a NAT and you don’t make sure that the server is able to resolve the sites dns to a ip it could actually connect to.
Since the server, behind the NAT, can’t connect it self using the “internet ip” your DNS records points to, request to the server itself will fail which might result in rather long timeouts.
This will happen whenever WordPress decides to call wp-cron.php to do some task, and even though the read-timeout for the wp-cron.php connection seems to be very low (something like 0.01s), the fact that the “internet ip” can not be connected at all seems to cause rather long timeouts.
I’m sure there is other situations where this might be an issue too, but the wp-cron-connection was the first and only I encountered.
The easiest way to fix it (if you’ve full system access) is probably to throw in a line pointing your domain to 127.0.0.1 in your servers hosts-file.
