remove_filter doesnt work

Hi!

In default-filters.php there is a Filter on the category-description that removes some HTML-Tags I like to keep.

So I try to do
remove_filter('pre_category_description', 'wp_filter_kses');
inside my Plugin-File

But this doesn’t work. I also tried adding priorities like 1, 10 and 30 like kackreiz suggested but this doesnt help.

I get the feeling the remove-mechanism does not work at all since all problems are solved when I uncomment the add_filter-line inside default-filters.php. A solution I dont like so much…

Is someone able to use the line of code above in his wordpress?
Whats wrong on my side?

Thanks!