Could I remove the icon font? (and use my own)

Since the attached Font-Awesome is such a big file:
is there a way to avoid using it (and loading it)?

I would like to replace the few I need with Fontello icons. I’m already using half a dozen of theirs, things that are missing in Font Awful (despite its size). And the nice thing with Fontello is that you can costomise it, so you only download the characters you need.

I only switched themes because mine is getting a bit bloated after too many updates and I would like to speed-up my site…

Thanks!