add_filter priority issue

Peeps,
I have a plugin that modifies links within posts, works fine. It is called like this :
add_filter('the_content', 'modifylinks');

I still have a problem however : I don’t want the plugin to modify the “Read more” link (the one generated by the_content on <!--more--->)
I thought tweaking my plugin priority would do, but whatever value I try it does not change the result (I thought setting it to 0 would make it, but obviously the_content is called before everyone else)

Any clue ?