How to include plugins just for one site?

In my blog i’m using some kind of gallery-plugin. On loading the blog, it will be included within the header for all sites and slow down site-laoding – although i need it just on one. Is there any possibility just to load it for one site?