wp_list_categories – show/hide count dynamically?

I’m using wp_list_categories to echo out a list of terms in a custom taxonomy (‘town’).

I want a different display depending on whether or not a term has posts assigned to it. So far I’ve added a class to the anchor tag of terms that have at least one post assigned so that I can highlight them in a different colour like so:

add_filter('wp_list_categories', 'custom_wp_list_categories');
function custom_wp_list_categories($list) {
	$cats = get_categories('taxonomy=town');
	foreach($cats as $cat) {
		$find = 'cat-item-' . $cat->term_id . '"';
		if($cat->count > 0){
		$replace = 'cat-item-' . $cat->term_id . ' active"';
		$list = str_replace( $find, $replace, $list );
		}
	}
	return $list;
}
wp_list_categories( 'taxonomy=town&hide_empty=0&show_count=1&title_li=' );

What I’d like to do now is display the post count for terms that have posts assigned to them, but display no post count for terms that don’t.

e.g.

no posts – “{term link}”
10 posts – “{term link} (10)”

I can’t really see a way to do it other than perhaps using jquery (not ideal) or stripping any instance of ” (0)” from the output of wp_list_categories – though I’m unsure how I would accomplish that.

Is it possible?