As I said, it is a personal thing – I just feel you can sometimes spend too long chasing the numbers. For me the important part is their recommendations and trying to implement as many as possible. And, yes, how the site REALLY looks and feels.
Also, you have to remember that some of these recommendations come with caveats as they have possible side effects – caching, for instance, often means that changes you may make to to a post or the site generally may not show up immediately (and if, like me, you forget that it’s down to the caching you can end up spending a lot of time scratching your head wondering what’s happened!).
I also remember a discussion on here recently where a user was being recommended by one of these sites to add a cache plugin but he found, when testing, that it added little to the end score and, in some cases, even made it slower (probably due to the overheads of the plugin running). I suspect this is because often the other recommendations (around changed to .htaccess) often add caching anyway.
So my recommendation – implement all the recommendations that you can but if it doesn’t improve your score then, well, probably don’t bother. When you’ve done all you can stop worrying about that end score – if your site feels quick and your users feel the same then you’ve done what you need to.
If you’re not already using it I would recommend the plugin GTmetrix for WordPress – from your admin it will get scores from PageSpeed and Yslow and give recommendations. I got mine to a ‘B’ score across the board and stopped there.