I have the same problem… running WordPress 3.1
Its adding approximately 2-3 seconds loading time to my website, even on pages where there are no star ratings being used.
I haven’t found a solution yet. But there is a workaround. If you go to the Graphics page in your settings panel, you can select specific sizes and types of stars/thumbs. Most likely you’re only going to be using one type of rating graphics right? If you unselect all the rest, they won’t load with the CSS file.