Robin Good: Although most website owners think that their web site is fine as it is, if you prompt them to go and check what Google Webmaster Tools says about their web site load times, they will generally remain surprised to discover that their site is slower than most others. And often by a large margin too.
Thus, if your speed test indicators inside Google Webmaster Tools say that your site is slower than 70% or more of the other sites out there, it is time to do something serious about this, as Google has clearly long announced that slow sites will be filtered, demoted and not given as much visibility as fast ones (that's what I'd still call a penalization).
Here is a great resource article that provides you with all of the info you need to check and verify whether you are doing the best to make your web site as fast as it can be.
Well done. Informative. Resourceful. 8/10
Full guide: http://www.seomoz.org/blog/15-tips-to-speed-up-your-website