I do not think that this is related to the plugin but rather to the performance of the browsers themselves. I just compared several sites containing (or not) the plugin, in Chrome (v15.0.874.121 m) and Firefox (v8.01) and there is a speed difference in all cases: I was surprised to see that Firefox was faster.