Remove tags from HTML comments only

Whenever I add an HTML comment into the text editor:

<!– comment here –>

Wordpress adds <p></p> tags before and after the comment, even if I strip out all spaces:

text<!– comment here –>text

I’ve tried this:

remove_filter( ‘the_content’, ‘wpautop’ );
remove_filter( ‘the_excerpt’, ‘wpautop’ );

It works great, except it removes the spaces throughout my site, and it’s too much work to go in every page to add them back in. Any thoughts for removing the <p> tags from comments only?